You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "TurboChargedDad ." <li...@gmail.com> on 2017/11/29 05:06:07 UTC
Understanding tomcat + apache and I/O
(Sorry didn't mean to send please check this email for additional info)
So.. Thank you for those help me understand the NIO vs BIO in tomcat 7.. I
made those changes things have improved quite a bit. I am still
experiencing some weirdness that I have tried to understand but can't get a
handle on it.
Quick overview..
--Proxies--
Apache Proxies (2) - The end user terminates SSL at the proxy/edge
The proxies use HTTPS/SSL to reverse proxy back to the tomcat server.
--/Proxies--
PXY1 & 2 configs for prefork mode.
<IfModule prefork.c>
StartServers 30
MinSpareServers 15
MaxSpareServers 30
ServerLimit 400
MaxClients 400
MaxRequestsPerChild 4000
</IfModule>
--Tomcat server-- (1)
Apache terminates SSL
over the top of Tomcat on the same server.
Reverse proxies to the tomcat server using NIO AJP connectors.
--/Tomcat server--
Tomcat apache prefork mode config:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 800
MaxClients 800
MaxRequestsPerChild 4000
</IfModule>
Typical vhost config for a given tenant would look like this..
<someuser.conf>
<VirtualHost 10.10.10.26:443>
ServerAdmin admin@company.com
ServerName somewhere.somedomain.com
ProxyPass / ajp://localhost:8126/ retry=3
DirectoryIndex index.php index.html index.htm
# if not specified, the global error log is used
ErrorLog "|/usr/sbin/rotatelogs /home/someuser/website/logs/
somewhere.somedomain.com-error_log_%Y%m%d 86400"
CustomLog "|/usr/sbin/rotatelogs /home/someuser/website/logs/
somewhere.somedomain.com-access_log_%Y%m%d 86400" combined
# log IP addresses
HostnameLookups Off
UseCanonicalName Off
ServerSignature off
SSLEngine on
SSLCertificateFile /etc/ssl/ssl.crt/somewhere.somedomain.com.crt
# Server Private Key:
SSLCertificateKeyFile /etc/ssl/ssl.key/somewhere.somedomain.com.key
SSLCertificateChainFile /etc/ssl/ssl.crt/somewhere.
somedomain-chain.com.crt
</VirtualHost>
</someuser.conf>
Typical tomcat connector thread config :
<Connector port="8126" protocol="org.apache.coyote.ajp.AjpNioProtocol"
redirectPort="8443" maxThreads="300" />
We are operating a multi-tenant environment. As of right now, we have
somewhere around 20 tomcat instances on a large machine of which only a
handful are "busy".
It used to be that when any one of them experienced a blocking issue.
Every one of them went down. All of their AJP connector threads would rise
until the system because tomcat was unresponsive. So far that appears for
the most part to be addressed...
However... When an issue is experienced. The site(s) experiencing the
issue(s) going down doesn't seem to bring down any of the other sites.
(w00t! w00t!)
But the httpd connections for each site all still climb together. (Please
see attached graph) Again no outage is experienced buy as demonstrated by
the graph attached to this message.
That graph is from zabbix using a custom metric that checks every 3 mins..
It does the following for each virtual host / tomcat instances
For user25 :
UserParameter=somewebsite.constats,sudo -tt /bin/netstat -ntp | grep EST |
grep httpd | grep ':8125' | wc -l
UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp | grep EST |
grep httpd | grep ':8126' | wc -l
So there is virtually no way they can be getting mixed up. Not to
mention that there are a few that do not experience a rise in connections.
Thoughts? Anyone?
Thanks in advance.
Re: Understanding tomcat + apache and I/O
Posted by "TurboChargedDad ." <li...@gmail.com>.
It should also be noted that if I bounce one of the larger instances.
Everyone suffers during the time to startup. The connection counts raise
in the same way although I am not sure at this time if there is an actual
outage experienced by anyone. I will have to do some testing to determine
that.
On Wed, Nov 29, 2017 at 2:16 PM, TurboChargedDad . <li...@gmail.com>
wrote:
> >> So now all you have to do is upgrade to Tomcat 8.0 or, even better,
> >>Tomcat 8.5 :)
> That's the plan but it's kind of like pulling teeth.
>
> >>Can you expand on the "weirdness"? I see you have some more details
> >>below but I think you could be more specific.
>
> Let's say that there are 12 users on a given system all running a tomcat
> server that has SSL terminated on the same host. user01 user02 user03 and
> son on all the way to user12. Each user has their own /home/userNN
> directory. Each user has their own own environment file in /etc/sysconfig/
> '/etc/sysconfig/tomcat7@userNN . In each of those files contains the
> various settings that are required for each user. CATALINA_HOME Java path,
> PID etc. Each user starts it's own JVM in a work directory in their home
> directory.
>
> Now imagine that user10's application starts to experience a database
> issue and the app stops responding.. It used to be true that everyone
> would stop responding because the AJP connectors were BIO. Then the HTTP
> connections would stack up across the board. The stacking of the HTTP
> connections was expected given the situation. Eventually the reverse proxy
> servers would die from running out of memory if were didn't get the outage
> under control quickly enough.
>
> Now that we switched that we have had 2 outages. In both cases the only
> tenants impacted from a performance perspective were the tenants
> experiencing the failures. No other alarms were detected during these
> outages for any other tenants. Something odd does happen however. The
> Apache HTTP connections rise for everyone along with the offending site.
>
> Please see the shared graph.
>
> https://photos.app.goo.gl/ZzEgpQUdbv9L84X82
>
> This is caclulated by doing a netstat and grepping for EST then httpd
> then the AJP port that would have connections passed back to it. ( sudo
> -tt > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l )
>
> tcp 0 0 127.0.0.1:37014 127.0.0.1:8125
> ESTABLISHED 5529/httpd
> tcp 0 0 127.0.0.1:40630 127.0.0.1:8125
> ESTABLISHED 29638/httpd
> tcp 0 0 127.0.0.1:40172 127.0.0.1:8125
> ESTABLISHED 28592/httpd
> tcp 0 0 127.0.0.1:36842 127.0.0.1:8125
> ESTABLISHED 5529/httpd
> tcp 0 0 127.0.0.1:40616 127.0.0.1:8125
> ESTABLISHED 29640/httpd
> tcp 0 0 127.0.0.1:37314 127.0.0.1:8125
> ESTABLISHED 20267/httpd
> tcp 0 0 127.0.0.1:39436 127.0.0.1:8125
> ESTABLISHED 29577/httpd
> tcp 0 0 127.0.0.1:39180 127.0.0.1:8125
> ESTABLISHED 25280/httpd
> tcp 0 0 127.0.0.1:40490 127.0.0.1:8125
> ESTABLISHED 29577/httpd
> tcp 0 0 127.0.0.1:39330 127.0.0.1:8125
> ESTABLISHED 29633/httpd
> tcp 0 0 127.0.0.1:40628 127.0.0.1:8125
> ESTABLISHED 29631/httpd
> tcp 0 0 127.0.0.1:39278 127.0.0.1:8125
> ESTABLISHED 28799/httpd
> tcp 0 0 127.0.0.1:39354 127.0.0.1:8125
> ESTABLISHED 29637/httpd
> tcp 0 0 127.0.0.1:39686 127.0.0.1:8125
> ESTABLISHED 29575/httpd
> tcp 0 0 127.0.0.1:37002 127.0.0.1:8125
> ESTABLISHED 8354/httpd
> tcp 0 0 127.0.0.1:39292 127.0.0.1:8125
> ESTABLISHED 29574/httpd
> tcp 0 0 127.0.0.1:39752 127.0.0.1:8125
> ESTABLISHED 29631/httpd
> tcp 0 0 127.0.0.1:41450 127.0.0.1:8125
> ESTABLISHED 29574/httpd
> tcp 0 0 127.0.0.1:37328 127.0.0.1:8125
> ESTABLISHED 20266/httpd
> tcp 0 0 127.0.0.1:39726 127.0.0.1:8125
> ESTABLISHED 28799/httpd
>
>
> It is the example above that determines the connection counts for each
> tenant.
>
> I cannot for the life of me understand how or why this is happening.. The
> only rise in connections should be detected in the offending application?
> Right?
>
> I can't say beyond a shadow of a doubt that the AJP connector threads
> aren't being wonky. I am having trouble getting JMX to tell me that
> information through zabbix.
>
>
> Thoughts?
>
> Thanks in advance.
>
> On Wed, Nov 29, 2017 at 8:51 AM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Big Papa,
>>
>> On 11/29/17 12:06 AM, TurboChargedDad . wrote:
>> > So.. Thank you for those help me understand the NIO vs BIO in
>> > tomcat 7..
>>
>> So now all you have to do is upgrade to Tomcat 8.0 or, even better,
>> Tomcat 8.5 :)
>>
>> > I made those changes things have improved quite a bit. I am still
>> > experiencing some weirdness that I have tried to understand but
>> > can't get a handle on it.
>>
>> Can you expand on the "weirdness"? I see you have some more details
>> below but I think you could be more specific.
>>
>> > Quick overview.. --Proxies-- Apache Proxies (2) - The end user
>> > terminates SSL at the proxy/edge The proxies use HTTPS/SSL to
>> > reverse proxy back to the tomcat server. --/Proxies--
>> >
>> > PXY1 & 2 configs for prefork mode. <IfModule prefork.c>
>> > StartServers 30 MinSpareServers 15 MaxSpareServers 30 ServerLimit
>> > 400 MaxClients 400 MaxRequestsPerChild 4000 </IfModule>
>>
>> If you want high performance, you have to abandon the prefork model
>> and move to event. Some modules (e.g. mod_php IIRC) don't work
>> properly with the event model. Think about using your lb with PHP
>> running on another server as Jim Riggs suggests[1]. You may get better
>> performance, stability, and fault-tolerance.
>>
>> > --Tomcat server-- (1) Apache terminates SSL over the top of Tomcat
>> > on the same server. Reverse proxies to the tomcat server using NIO
>> > AJP connectors. --/Tomcat server--
>>
>> Above you say that you are using HTTPS/SSL to connect httpd -> Tomcat.
>> If you are using AJP then this is not true. So which is it? Are you
>> using HTTP or AJP as your protocol?
>>
>> > Tomcat apache prefork mode config: <IfModule prefork.c>
>> > StartServers 8 MinSpareServers 5 MaxSpareServers 20
>> > ServerLimit 800 MaxClients 800 MaxRequestsPerChild
>> > 4000 </IfModule>
>>
>> What does "Tomcat apache prefork mode" mean? The above is an httpd
>> configuration, not a Tomcat one.
>>
>> > Typical vhost config for a given tenant would look like this..
>> > <someuser.conf> <VirtualHost 10.10.10.26:443
>> > <http://10.10.10.26:443/>> ServerAdmin admin@company.com
>> > <ma...@company.com> ServerName somewhere.somedomain.com
>> > <http://somewhere.somedomain.com/> ProxyPass /
>> > ajp://localhost:8126/ retry=3
>>
>> Okay, now you are using AJP. I think there's definitely some confusion
>> here as to what is being configured with what.
>>
>> > Typical tomcat connector thread config : <Connector port="8126"
>> > protocol="org.apache.coyote.ajp.AjpNioProtocol"
>> > redirectPort="8443" maxThreads="300" />
>>
>> If this is the only <Connector> in Tomcat, then you are 100% using AJP
>> and not HTTP as your protocol.
>>
>> Using NIO is the best practice here IMO.
>>
>> > We are operating a multi-tenant environment. As of right now, we
>> > have somewhere around 20 tomcat instances on a large machine of
>> > which only a handful are "busy".
>>
>> Good.
>>
>> > It used to be that when any one of them experienced a blocking
>> > issue. Every one of them went down. All of their AJP connector
>> > threads would rise until the system because tomcat was
>> > unresponsive.
>>
>> That would be a capacity-planning problem with the httpd proxies. You
>> probably didn't do your math correctly.
>>
>> > So far that appears for the most part to be addressed...
>> Good. Maybe your math is better but it may still be wrong.
>>
>> > However... When an issue is experienced. The site(s)
>> > experiencing the issue(s) going down doesn't seem to bring down any
>> > of the other sites. (w00t! w00t!)
>>
>> Good.
>>
>> > But the httpd connections for each site all still climb together.
>>
>> That shouldn't happen (of course!).
>>
>> > (Please see attached graph) Again no outage is experienced buy as
>> > demonstrated by the graph attached to this message.
>>
>> Attachments are stripped. Either post your graph elsewhere or describe
>> it in words.
>>
>> > That graph is from zabbix using a custom metric that checks every
>> > 3 mins.. It does the following for each virtual host / tomcat
>> > instances
>> >
>> > For user25 : UserParameter=somewebsite.constats,sudo -tt
>> > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l
>> > UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp |
>> > grep EST | grep httpd | grep ':8126' | wc -l
>> >
>> > So there is virtually no way they can be getting mixed up. Not to
>> > mention that there are a few that do not experience a rise in
>> > connections.
>>
>> So the "Weirdness" is that your AJP connection count on the httpd
>> proxy instances increases across all web servers (or all workers?).
>> What does mod_proxy's status page say for *each worker*? THAT'S what
>> you need to compare, not just the total number of connections/threads
>> on the proxy.
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloeyX0dHGNocmlzQGNo
>> cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgAZQ/+OyyDIEaWzgF5zG1o
>> amUGjCUackktehlpW9STa5kRhIj9REYT4Cql64Cwqvw8ciZVQXAOsYJBACXFKcfa
>> fvegRQ03YeLy9LDXhPtsx4Nr+qT17ySiFo/MckEIkxCR9mBbFokUb1bVes9kkYQu
>> yJjQ7AV8SWDWKGdAkbRk4WTuJ23bvRwZ2g4MNb0sDg5dJEQIOY7JYhlFJQLPm/1a
>> Yeeo/xRMLfY4FBI0zpA1DAXEwiLyXup4SOztHnoxbK5h0YgrRGMOKvAwZXs5/u/2
>> NbiqCnsA80OzUrSXd5sDBYzsuR2yOfnnUMcUJh6LmL8XG1Fh+QeJdCDM2KCC0hJL
>> HSyqEVfl9EyIXj/Bu0DzU6lL8z5lhxoEYeYBE+pMfJKZ3Skb3EHLImuI/Fwv/+hk
>> T13o1irPIiqq0N0pTvjLtbJbapNZ9Nz8aIzTBLR3TRY6Cf79jI49QRHBYMsth7e/
>> 2Ub+WmkbphcoaC1oJEpXu8fbcSuDrdTAzZ9VSvUFsqpUw5b/9OSB2DjKx5GtnHzR
>> f8bEg682xw8VKlOV7EsD4RvyILmFqisrrTAfE8/3F3IGMYqXV8Is+8BtdScvUgT5
>> MzaLVVyT650pv1NJxBjz/UF8BdhJ2Rpj0r5jhqB64Q1F1ZuNO6aLI2qEwRNgwYMV
>> IoMgNCcqcB03pYbpuYxHKoX5Cfc=
>> =imbH
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: Understanding tomcat + apache and I/O
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
To whom it may concern,
On 11/29/17 3:16 PM, TurboChargedDad . wrote:
>>> So now all you have to do is upgrade to Tomcat 8.0 or, even
>>> better, Tomcat 8.5 :)
> That's the plan but it's kind of like pulling teeth.
>
>>> Can you expand on the "weirdness"? I see you have some more
>>> details below but I think you could be more specific.
>
> Let's say that there are 12 users on a given system all running a
> tomcat server that has SSL terminated on the same host. user01
> user02 user03 and son on all the way to user12. Each user has
> their own /home/userNN directory. Each user has their own own
> environment file in /etc/sysconfig/ '/etc/sysconfig/tomcat7@userNN
> . In each of those files contains the various settings that are
> required for each user. CATALINA_HOME Java path, PID etc. Each
> user starts it's own JVM in a work directory in their home
> directory.
So... separate Tomcat processes for each user, right?
> Now imagine that user10's application starts to experience a
> database issue and the app stops responding.. It used to be true
> that everyone would stop responding because the AJP connectors were
> BIO.
That doesn't make any sense: the proxy configuration is much more
important than whether or not BIO is being used with Tomcat. If one
Tomcat is seized-up (which would happen e.g. if all db connections
were used and all BIO threads were blocking), there should be no
problem with the other workers on the proxy reaching other Tomcat
instances.
> Then the HTTP connections would stack up across the board. The
> stacking of the HTTP connections was expected given the situation.
> Eventually the reverse proxy servers would die from running out of
> memory if were didn't get the outage under control quickly enough.
Now, /that/ I can understand.
But you said you were using prefork. :/
> Now that we switched that we have had 2 outages. In both cases the
> only tenants impacted from a performance perspective were the
> tenants experiencing the failures.
Good: if an application fails, it should expect to suffer downtime,
but it shouldn't affect the other applications. Assuming we are
talking about separate Tomcat JVM instances.
> No other alarms were detected during these outages for any other
> tenants. Something odd does happen however. The Apache HTTP
> connections rise for everyone along with the offending site.
> Please see the shared graph.
>
> https://photos.app.goo.gl/ZzEgpQUdbv9L84X82
>
> This is caclulated by doing a netstat and grepping for EST then
> httpd then the AJP port that would have connections passed back to
> it. ( sudo -tt > /bin/netstat -ntp | grep EST | grep httpd | grep
> ':8125' | wc -l )
So what's :8125? The failing application or one of the good ones?
If httpd is over-provisioned (which is usually okay) and Tomcat is
stalled, then the number of hung connections from httpd -> Tomcat
rising is exactly what I'd expect: the web server can accept more
connections so it's doing that. Tomcat can't make progress so everyone
waits.
> It is the example above that determines the connection counts for
> each tenant.
>
> I cannot for the life of me understand how or why this is
> happening.. The only rise in connections should be detected in the
> offending application? Right?
Yes. What you've shown me is that a failing application experiences
high connection counts which makes sense. Unless I'm reading your text
above incorrectly. Or the graph?
> I can't say beyond a shadow of a doubt that the AJP connector
> threads aren't being wonky. I am having trouble getting JMX to
> tell me that information through zabbix.
How are you querying JMX?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlogOnwdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFignBAAjU3tpDu/4YqpMOAy
kPKgD9/U8gWBuxJpDbEbMwErPjv0gFGGwYKz/VE0d6of+DEJtVn/ZQKaaTSHJ+f1
FwHmJLYr+H8GI3Nc9dobDOtgE5fyEekqWByHA6r+4jU3hBWjqYzcT+ihcEhHhs1v
xy3cOaUd/4G/XON9HtoxsQ3wdiOlBT06GNQpYQNJl3/zlYrS+5I4qoYIazQ21dq9
qpNVRuRID5ltIjbq/G95S+y+srMW2xX7O7XUz520gc546C8p+29hmppEoUTjyxch
wVn46UEAkIbWaCOCCAYikUlwLPMPLhBhCE/vldCqjqq1VMtPV5fhhLUxm/kNac26
mvIFZuCrAv2jD+SDpB+trQK696STDYlCNrn3s3mMw+fkWVnIhiI3OZPXDzieD+Bf
MXO26uAMwAnjsWuJ6M8u+OGhufWW6YetVpJZpibJWCOcx5NLDkm+7BReeVEn1Vv4
p5zjXT/rePAlhz3T+7oUCCUN/O+X7BPIcjdeoh8mp+W3t30dYWWQJI4hHt+ATuq1
7J1fUzyjJNE7iP5kvxnG5lA3Kc0clobhwyMH8D5dgqwR32Hm+7IPE2PQbtx4SU8O
pqNTIdi2l2h1TTAjmey1TtP85GZHsl6Rwy0gP8nRso4UtGZp5EPPsUhrRfPSS85D
MD3low32ZefGl1Omb1lDCFSEbDg=
=qCJH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Understanding tomcat + apache and I/O
Posted by "TurboChargedDad ." <li...@gmail.com>.
>> So now all you have to do is upgrade to Tomcat 8.0 or, even better,
>>Tomcat 8.5 :)
That's the plan but it's kind of like pulling teeth.
>>Can you expand on the "weirdness"? I see you have some more details
>>below but I think you could be more specific.
Let's say that there are 12 users on a given system all running a tomcat
server that has SSL terminated on the same host. user01 user02 user03 and
son on all the way to user12. Each user has their own /home/userNN
directory. Each user has their own own environment file in /etc/sysconfig/
'/etc/sysconfig/tomcat7@userNN . In each of those files contains the
various settings that are required for each user. CATALINA_HOME Java path,
PID etc. Each user starts it's own JVM in a work directory in their home
directory.
Now imagine that user10's application starts to experience a database
issue and the app stops responding.. It used to be true that everyone
would stop responding because the AJP connectors were BIO. Then the HTTP
connections would stack up across the board. The stacking of the HTTP
connections was expected given the situation. Eventually the reverse proxy
servers would die from running out of memory if were didn't get the outage
under control quickly enough.
Now that we switched that we have had 2 outages. In both cases the only
tenants impacted from a performance perspective were the tenants
experiencing the failures. No other alarms were detected during these
outages for any other tenants. Something odd does happen however. The
Apache HTTP connections rise for everyone along with the offending site.
Please see the shared graph.
https://photos.app.goo.gl/ZzEgpQUdbv9L84X82
This is caclulated by doing a netstat and grepping for EST then httpd
then the AJP port that would have connections passed back to it. ( sudo -tt >
/bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l )
tcp 0 0 127.0.0.1:37014 127.0.0.1:8125
ESTABLISHED 5529/httpd
tcp 0 0 127.0.0.1:40630 127.0.0.1:8125
ESTABLISHED 29638/httpd
tcp 0 0 127.0.0.1:40172 127.0.0.1:8125
ESTABLISHED 28592/httpd
tcp 0 0 127.0.0.1:36842 127.0.0.1:8125
ESTABLISHED 5529/httpd
tcp 0 0 127.0.0.1:40616 127.0.0.1:8125
ESTABLISHED 29640/httpd
tcp 0 0 127.0.0.1:37314 127.0.0.1:8125
ESTABLISHED 20267/httpd
tcp 0 0 127.0.0.1:39436 127.0.0.1:8125
ESTABLISHED 29577/httpd
tcp 0 0 127.0.0.1:39180 127.0.0.1:8125
ESTABLISHED 25280/httpd
tcp 0 0 127.0.0.1:40490 127.0.0.1:8125
ESTABLISHED 29577/httpd
tcp 0 0 127.0.0.1:39330 127.0.0.1:8125
ESTABLISHED 29633/httpd
tcp 0 0 127.0.0.1:40628 127.0.0.1:8125
ESTABLISHED 29631/httpd
tcp 0 0 127.0.0.1:39278 127.0.0.1:8125
ESTABLISHED 28799/httpd
tcp 0 0 127.0.0.1:39354 127.0.0.1:8125
ESTABLISHED 29637/httpd
tcp 0 0 127.0.0.1:39686 127.0.0.1:8125
ESTABLISHED 29575/httpd
tcp 0 0 127.0.0.1:37002 127.0.0.1:8125
ESTABLISHED 8354/httpd
tcp 0 0 127.0.0.1:39292 127.0.0.1:8125
ESTABLISHED 29574/httpd
tcp 0 0 127.0.0.1:39752 127.0.0.1:8125
ESTABLISHED 29631/httpd
tcp 0 0 127.0.0.1:41450 127.0.0.1:8125
ESTABLISHED 29574/httpd
tcp 0 0 127.0.0.1:37328 127.0.0.1:8125
ESTABLISHED 20266/httpd
tcp 0 0 127.0.0.1:39726 127.0.0.1:8125
ESTABLISHED 28799/httpd
It is the example above that determines the connection counts for each
tenant.
I cannot for the life of me understand how or why this is happening.. The
only rise in connections should be detected in the offending application?
Right?
I can't say beyond a shadow of a doubt that the AJP connector threads
aren't being wonky. I am having trouble getting JMX to tell me that
information through zabbix.
Thoughts?
Thanks in advance.
On Wed, Nov 29, 2017 at 8:51 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Big Papa,
>
> On 11/29/17 12:06 AM, TurboChargedDad . wrote:
> > So.. Thank you for those help me understand the NIO vs BIO in
> > tomcat 7..
>
> So now all you have to do is upgrade to Tomcat 8.0 or, even better,
> Tomcat 8.5 :)
>
> > I made those changes things have improved quite a bit. I am still
> > experiencing some weirdness that I have tried to understand but
> > can't get a handle on it.
>
> Can you expand on the "weirdness"? I see you have some more details
> below but I think you could be more specific.
>
> > Quick overview.. --Proxies-- Apache Proxies (2) - The end user
> > terminates SSL at the proxy/edge The proxies use HTTPS/SSL to
> > reverse proxy back to the tomcat server. --/Proxies--
> >
> > PXY1 & 2 configs for prefork mode. <IfModule prefork.c>
> > StartServers 30 MinSpareServers 15 MaxSpareServers 30 ServerLimit
> > 400 MaxClients 400 MaxRequestsPerChild 4000 </IfModule>
>
> If you want high performance, you have to abandon the prefork model
> and move to event. Some modules (e.g. mod_php IIRC) don't work
> properly with the event model. Think about using your lb with PHP
> running on another server as Jim Riggs suggests[1]. You may get better
> performance, stability, and fault-tolerance.
>
> > --Tomcat server-- (1) Apache terminates SSL over the top of Tomcat
> > on the same server. Reverse proxies to the tomcat server using NIO
> > AJP connectors. --/Tomcat server--
>
> Above you say that you are using HTTPS/SSL to connect httpd -> Tomcat.
> If you are using AJP then this is not true. So which is it? Are you
> using HTTP or AJP as your protocol?
>
> > Tomcat apache prefork mode config: <IfModule prefork.c>
> > StartServers 8 MinSpareServers 5 MaxSpareServers 20
> > ServerLimit 800 MaxClients 800 MaxRequestsPerChild
> > 4000 </IfModule>
>
> What does "Tomcat apache prefork mode" mean? The above is an httpd
> configuration, not a Tomcat one.
>
> > Typical vhost config for a given tenant would look like this..
> > <someuser.conf> <VirtualHost 10.10.10.26:443
> > <http://10.10.10.26:443/>> ServerAdmin admin@company.com
> > <ma...@company.com> ServerName somewhere.somedomain.com
> > <http://somewhere.somedomain.com/> ProxyPass /
> > ajp://localhost:8126/ retry=3
>
> Okay, now you are using AJP. I think there's definitely some confusion
> here as to what is being configured with what.
>
> > Typical tomcat connector thread config : <Connector port="8126"
> > protocol="org.apache.coyote.ajp.AjpNioProtocol"
> > redirectPort="8443" maxThreads="300" />
>
> If this is the only <Connector> in Tomcat, then you are 100% using AJP
> and not HTTP as your protocol.
>
> Using NIO is the best practice here IMO.
>
> > We are operating a multi-tenant environment. As of right now, we
> > have somewhere around 20 tomcat instances on a large machine of
> > which only a handful are "busy".
>
> Good.
>
> > It used to be that when any one of them experienced a blocking
> > issue. Every one of them went down. All of their AJP connector
> > threads would rise until the system because tomcat was
> > unresponsive.
>
> That would be a capacity-planning problem with the httpd proxies. You
> probably didn't do your math correctly.
>
> > So far that appears for the most part to be addressed...
> Good. Maybe your math is better but it may still be wrong.
>
> > However... When an issue is experienced. The site(s)
> > experiencing the issue(s) going down doesn't seem to bring down any
> > of the other sites. (w00t! w00t!)
>
> Good.
>
> > But the httpd connections for each site all still climb together.
>
> That shouldn't happen (of course!).
>
> > (Please see attached graph) Again no outage is experienced buy as
> > demonstrated by the graph attached to this message.
>
> Attachments are stripped. Either post your graph elsewhere or describe
> it in words.
>
> > That graph is from zabbix using a custom metric that checks every
> > 3 mins.. It does the following for each virtual host / tomcat
> > instances
> >
> > For user25 : UserParameter=somewebsite.constats,sudo -tt
> > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l
> > UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp |
> > grep EST | grep httpd | grep ':8126' | wc -l
> >
> > So there is virtually no way they can be getting mixed up. Not to
> > mention that there are a few that do not experience a rise in
> > connections.
>
> So the "Weirdness" is that your AJP connection count on the httpd
> proxy instances increases across all web servers (or all workers?).
> What does mod_proxy's status page say for *each worker*? THAT'S what
> you need to compare, not just the total number of connections/threads
> on the proxy.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloeyX0dHGNocmlzQGNo
> cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgAZQ/+OyyDIEaWzgF5zG1o
> amUGjCUackktehlpW9STa5kRhIj9REYT4Cql64Cwqvw8ciZVQXAOsYJBACXFKcfa
> fvegRQ03YeLy9LDXhPtsx4Nr+qT17ySiFo/MckEIkxCR9mBbFokUb1bVes9kkYQu
> yJjQ7AV8SWDWKGdAkbRk4WTuJ23bvRwZ2g4MNb0sDg5dJEQIOY7JYhlFJQLPm/1a
> Yeeo/xRMLfY4FBI0zpA1DAXEwiLyXup4SOztHnoxbK5h0YgrRGMOKvAwZXs5/u/2
> NbiqCnsA80OzUrSXd5sDBYzsuR2yOfnnUMcUJh6LmL8XG1Fh+QeJdCDM2KCC0hJL
> HSyqEVfl9EyIXj/Bu0DzU6lL8z5lhxoEYeYBE+pMfJKZ3Skb3EHLImuI/Fwv/+hk
> T13o1irPIiqq0N0pTvjLtbJbapNZ9Nz8aIzTBLR3TRY6Cf79jI49QRHBYMsth7e/
> 2Ub+WmkbphcoaC1oJEpXu8fbcSuDrdTAzZ9VSvUFsqpUw5b/9OSB2DjKx5GtnHzR
> f8bEg682xw8VKlOV7EsD4RvyILmFqisrrTAfE8/3F3IGMYqXV8Is+8BtdScvUgT5
> MzaLVVyT650pv1NJxBjz/UF8BdhJ2Rpj0r5jhqB64Q1F1ZuNO6aLI2qEwRNgwYMV
> IoMgNCcqcB03pYbpuYxHKoX5Cfc=
> =imbH
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Understanding tomcat + apache and I/O
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Big Papa,
On 11/29/17 12:06 AM, TurboChargedDad . wrote:
> So.. Thank you for those help me understand the NIO vs BIO in
> tomcat 7..
So now all you have to do is upgrade to Tomcat 8.0 or, even better,
Tomcat 8.5 :)
> I made those changes things have improved quite a bit. I am still
> experiencing some weirdness that I have tried to understand but
> can't get a handle on it.
Can you expand on the "weirdness"? I see you have some more details
below but I think you could be more specific.
> Quick overview.. --Proxies-- Apache Proxies (2) - The end user
> terminates SSL at the proxy/edge The proxies use HTTPS/SSL to
> reverse proxy back to the tomcat server. --/Proxies--
>
> PXY1 & 2 configs for prefork mode. <IfModule prefork.c>
> StartServers 30 MinSpareServers 15 MaxSpareServers 30 ServerLimit
> 400 MaxClients 400 MaxRequestsPerChild 4000 </IfModule>
If you want high performance, you have to abandon the prefork model
and move to event. Some modules (e.g. mod_php IIRC) don't work
properly with the event model. Think about using your lb with PHP
running on another server as Jim Riggs suggests[1]. You may get better
performance, stability, and fault-tolerance.
> --Tomcat server-- (1) Apache terminates SSL over the top of Tomcat
> on the same server. Reverse proxies to the tomcat server using NIO
> AJP connectors. --/Tomcat server--
Above you say that you are using HTTPS/SSL to connect httpd -> Tomcat.
If you are using AJP then this is not true. So which is it? Are you
using HTTP or AJP as your protocol?
> Tomcat apache prefork mode config: <IfModule prefork.c>
> StartServers 8 MinSpareServers 5 MaxSpareServers 20
> ServerLimit 800 MaxClients 800 MaxRequestsPerChild
> 4000 </IfModule>
What does "Tomcat apache prefork mode" mean? The above is an httpd
configuration, not a Tomcat one.
> Typical vhost config for a given tenant would look like this..
> <someuser.conf> <VirtualHost 10.10.10.26:443
> <http://10.10.10.26:443/>> ServerAdmin admin@company.com
> <ma...@company.com> ServerName somewhere.somedomain.com
> <http://somewhere.somedomain.com/> ProxyPass /
> ajp://localhost:8126/ retry=3
Okay, now you are using AJP. I think there's definitely some confusion
here as to what is being configured with what.
> Typical tomcat connector thread config : <Connector port="8126"
> protocol="org.apache.coyote.ajp.AjpNioProtocol"
> redirectPort="8443" maxThreads="300" />
If this is the only <Connector> in Tomcat, then you are 100% using AJP
and not HTTP as your protocol.
Using NIO is the best practice here IMO.
> We are operating a multi-tenant environment. As of right now, we
> have somewhere around 20 tomcat instances on a large machine of
> which only a handful are "busy".
Good.
> It used to be that when any one of them experienced a blocking
> issue. Every one of them went down. All of their AJP connector
> threads would rise until the system because tomcat was
> unresponsive.
That would be a capacity-planning problem with the httpd proxies. You
probably didn't do your math correctly.
> So far that appears for the most part to be addressed...
Good. Maybe your math is better but it may still be wrong.
> However... When an issue is experienced. The site(s)
> experiencing the issue(s) going down doesn't seem to bring down any
> of the other sites. (w00t! w00t!)
Good.
> But the httpd connections for each site all still climb together.
That shouldn't happen (of course!).
> (Please see attached graph) Again no outage is experienced buy as
> demonstrated by the graph attached to this message.
Attachments are stripped. Either post your graph elsewhere or describe
it in words.
> That graph is from zabbix using a custom metric that checks every
> 3 mins.. It does the following for each virtual host / tomcat
> instances
>
> For user25 : UserParameter=somewebsite.constats,sudo -tt
> /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l
> UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp |
> grep EST | grep httpd | grep ':8126' | wc -l
>
> So there is virtually no way they can be getting mixed up. Not to
> mention that there are a few that do not experience a rise in
> connections.
So the "Weirdness" is that your AJP connection count on the httpd
proxy instances increases across all web servers (or all workers?).
What does mod_proxy's status page say for *each worker*? THAT'S what
you need to compare, not just the total number of connections/threads
on the proxy.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloeyX0dHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgAZQ/+OyyDIEaWzgF5zG1o
amUGjCUackktehlpW9STa5kRhIj9REYT4Cql64Cwqvw8ciZVQXAOsYJBACXFKcfa
fvegRQ03YeLy9LDXhPtsx4Nr+qT17ySiFo/MckEIkxCR9mBbFokUb1bVes9kkYQu
yJjQ7AV8SWDWKGdAkbRk4WTuJ23bvRwZ2g4MNb0sDg5dJEQIOY7JYhlFJQLPm/1a
Yeeo/xRMLfY4FBI0zpA1DAXEwiLyXup4SOztHnoxbK5h0YgrRGMOKvAwZXs5/u/2
NbiqCnsA80OzUrSXd5sDBYzsuR2yOfnnUMcUJh6LmL8XG1Fh+QeJdCDM2KCC0hJL
HSyqEVfl9EyIXj/Bu0DzU6lL8z5lhxoEYeYBE+pMfJKZ3Skb3EHLImuI/Fwv/+hk
T13o1irPIiqq0N0pTvjLtbJbapNZ9Nz8aIzTBLR3TRY6Cf79jI49QRHBYMsth7e/
2Ub+WmkbphcoaC1oJEpXu8fbcSuDrdTAzZ9VSvUFsqpUw5b/9OSB2DjKx5GtnHzR
f8bEg682xw8VKlOV7EsD4RvyILmFqisrrTAfE8/3F3IGMYqXV8Is+8BtdScvUgT5
MzaLVVyT650pv1NJxBjz/UF8BdhJ2Rpj0r5jhqB64Q1F1ZuNO6aLI2qEwRNgwYMV
IoMgNCcqcB03pYbpuYxHKoX5Cfc=
=imbH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org