You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Suvendu Sekhar Mondal <su...@gmail.com> on 2018/01/17 13:20:37 UTC
Which Connector properties will be used in case of redirection?
Hello Everyone,
I am seeing one issue. Under high load sporadically one web service
call fails with: "java.net.SocketTimeoutException: Read timed out"
after 60 Sec mark.
In our app, httpd routes requests to 8888 then it get redirected to
8889. Connector which listens on 8888 has connectionTimeout=20000 but
the one which listens to 8889 does not have any connectionTimeout set
explicitly. As per doc, default connectionTimeout value 60 Sec will be
set for connector which listens on 8889 - and that is playing a role
here. Do you think this assumption is correct?
Here are the connector properties:
<Connector port="8888" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8889" />
<Connector
port="8889"
SSLCertificateFile="C:/mycer.cer"
SSLCertificateKeyFile="C:\mykey.key"
SSLEnabled="true"
acceptCount="100"
clientAuth="false"
disableUploadTimeout="true"
enableLookups="false"
keystoreFile="conf/.keystore"
maxHttpHeaderSize="8192"
maxSavePostSize="-1"
maxThreads="200"
minSpareThreads="20"
protocol="HTTP/1.1"
scheme="https"
secure="true"
sslProtocol="TLS1.2"
compressableMimeTypes="text/html,text/xml,text/plain,application/json"
compressionMinSize="2048"
compression="force"
threadPriority="6" />
Thanks!
Suvendu
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Which Connector properties will be used in case of redirection?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Suvendu,
On 1/18/18 2:55 AM, Suvendu Sekhar Mondal wrote:
> Hi Christopher,
>
> On Wed, Jan 17, 2018 at 10:41 PM, Christopher Schultz
> <ch...@christopherschultz.net> wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> Suvendu,
>>
>> Which version of Tomcat?
>>
>
> Tomcat version is 7.0.55
>
>> On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote:
>>> I am seeing one issue. Under high load sporadically one web
>>> service call fails with: "java.net.SocketTimeoutException:
>>> Read timed out" after 60 Sec mark.
>>>
>>> In our app, httpd routes requests to 8888 then it get
>>> redirected to 8889.
>>
>> Can you be very explicit about what you mean by "redirect",
>> here?
>
> I was talking about auto redirection from 8888 to 8889 by
> Catalina.
>
>>> Connector which listens on 8888 has connectionTimeout=20000 but
>>> the one which listens to 8889 does not have any
>>> connectionTimeout set explicitly. As per doc, default
>>> connectionTimeout value 60 Sec will be set for connector which
>>> listens on 8889 - and that is playing a role here. Do you think
>>> this assumption is correct?
>>>
>>> Here are the connector properties: <Connector port="8888"
>>> protocol="HTTP/1.1" connectionTimeout="20000"
>>> redirectPort="8889" />
>>
>> So you are probably expecting that all HTTP traffic to :8888 will
>> be redirected to HTTPS over :8889?
>>
>
> Yes.
>
>> How long does the redirect take?
>>
>
> I do not know how to measure redirection time. Please let me know
> if there any mechanism to trace it.
>
>> Under heavy load, you might be running out of threads depending
>> upon a lot of factors, such as what kind of protocol and endpoint
>> are actually being used. It looks like you are using the APR
>> connector (due to SSLCertificateFile) which has some positive
>> attributes with request to scalability while NIO would probably
>> be the best choice if possible.
>>
>> It's got terrible performance, though, unless you use the
>> OpenSSL provider (which requires Tomcat 8.5 or later).
>>
>>> <Connector port="8889" SSLCertificateFile="C:/mycer.cer"
>>> SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true"
>>> acceptCount="100" clientAuth="false"
>>> disableUploadTimeout="true" enableLookups="false"
>>> keystoreFile="conf/.keystore" maxHttpHeaderSize="8192"
>>> maxSavePostSize="-1" maxThreads="200" minSpareThreads="20"
>>> protocol="HTTP/1.1" scheme="https" secure="true"
>>> sslProtocol="TLS1.2"
>>> compressableMimeTypes="text/html,text/xml,text/plain,application/jso
n"
>>>
>>>
>>
>>>
compressionMinSize="2048"
>>> compression="force" threadPriority="6" />
>>
>> When you say "high load", what kind of load are you talking
>> about, specifically?
>>
>> You may simply have too much traffic for your existing hardware
>> to handle. No amount of configuration can fix that. - -chris
>
> We have 9 JVMs. As per current setting, per JVM max concurrency
> can reach up-to 300(maxThreads="200"+acceptCount="100"). Issue was
> reported when 2300 users on the system.
So that's 2300 users across all 9 JVMs with the same configuration?
Your cluster ought to be able to support 9 * 200 = 1800 simultaneous
active users plus 9 * 100 = 900 queued in the OS's TCP wait queue.
2300 should be within your capabilities, but it might depend heavily
upon how the load-balancer decided to allocate requests. If it sent
all the users to the same node in the cluster, then you'll see bad
performance even if you "should" be able to handle that much load.
Where is the SocketTimeoutException being thrown? On the server, or on
the client? If it's on the server, then it's the client's fault (or
the load-balancer's fault) and it's not a problem at all with your
server capacity.
If it's happening on the client, it makes a lot more sense that the
server might be overloaded. But that 60-second timeout would have to
be on the client, not on the server side. So I'm a little confused as
to what is going on, here.
> So, I think it is not a case of thread exhaustion but something
> else. Most possibly few JVMs were under stress due to uneven load
> balancing(we have stickysession). This issue just came to me and
> all resource usage data are gone. I have asked for re-run so that I
> can analyze it properly. Let's see. :)
It would be good to see what the session-load and the current-user
(meaning in-process request) load was on each node in the cluster at
the time, and which of them seems to be failing.
> Back to my original question, in case of HTTP to HTTPS auto
> redirection, Connector properties set for the later will be
> finally applied to the request - is that a correct statement?
The order of events goes like this:
1. Client connects to server via HTTP
2. Server redirects client to HTTPS port [1]
3. Client connects to server via HTTPS
2. Server responds appropriately[2]
[1] here is the 20s connect timeout, which isn't actually a connection
timeout... it's a timeout waiting for the client to present the HTTP
request line before terminating the connection.
[2] here is the 60s connect timeout [which isn't a connection timeout,
of course].
Once that second connection is made (the HTTPS connection), the 20s
connect-timeout is no longer applicable.
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpgtisdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFg9thAAhv8j+ioCuJD2Vr/e
ovZEWkFKVhl1yWD3YTUX0KoTGot1MYp096jlU3BWaTBNaNxb9ay2U1ar4WLtRfky
9SFVaDEKykKN88yBCAuj6LCFtqJq5kxk8jGnbvjeN66TEaOOCQaGZNHu/JxPiCz4
UI82ynx5HFVM0fvhHublyv8+GdaZvjZa0mDYcp/1p0ySwqlyD3gy7eBk04z3JDan
GZ9DLdshG/YtIeMT8VmBae565PKOs8WGaxMoj+/tnbnCmPNLu1CzgoBuQDwCu7vp
mKQMYPCsnX4Hfi9RVoa8d8porOi2TxaJcpF3KFTujGSFebzcRHEZh1DHWP7ugfcb
3Zx2WAYD9Mbf0k3INoBROpfG/nDHtqgMLVfPRmBEuX11QluB7b1uP73XLfviyg7V
JeEsCrb/8lXGC9j9xS9wc02ozLZb0KI/AdHSBiu673Q0r5IEMTgBM7pLiuIyMT5U
4o58c1VtkqGuhgf0vOCDyxYoSM4IOfK+DpOvWbvByOjhNVAKHJObgBB7koF8g+xQ
bIsr9P30zUukE6Er01/jGyfhMCZJP5HWot7mDM2RFLpseqSu2g9NGLkFCJiNKRpO
uBOXh8mCq/LPuAHRqSmNKnNOxWmQ6/5I0yaXRF85KP4J4cXRI2037+90Pp+37QsV
qOtrMzDtuE0x/Wlp17ARe38fkHw=
=5NhI
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Which Connector properties will be used in case of redirection?
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hi Christopher,
On Wed, Jan 17, 2018 at 10:41 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Suvendu,
>
> Which version of Tomcat?
>
Tomcat version is 7.0.55
> On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote:
>> I am seeing one issue. Under high load sporadically one web
>> service call fails with: "java.net.SocketTimeoutException: Read
>> timed out" after 60 Sec mark.
>>
>> In our app, httpd routes requests to 8888 then it get redirected
>> to 8889.
>
> Can you be very explicit about what you mean by "redirect", here?
I was talking about auto redirection from 8888 to 8889 by Catalina.
>> Connector which listens on 8888 has connectionTimeout=20000 but the
>> one which listens to 8889 does not have any connectionTimeout set
>> explicitly. As per doc, default connectionTimeout value 60 Sec will
>> be set for connector which listens on 8889 - and that is playing a
>> role here. Do you think this assumption is correct?
>>
>> Here are the connector properties: <Connector port="8888"
>> protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8889"
>> />
>
> So you are probably expecting that all HTTP traffic to :8888 will be
> redirected to HTTPS over :8889?
>
Yes.
> How long does the redirect take?
>
I do not know how to measure redirection time. Please let me know if
there any mechanism to trace it.
> Under heavy load, you might be running out of threads depending upon a
> lot of factors, such as what kind of protocol and endpoint are
> actually being used. It looks like you are using the APR connector
> (due to SSLCertificateFile) which has some positive attributes with
> request to scalability while NIO would probably be the best choice if
> possible.
>
> It's got terrible performance, though, unless you use the OpenSSL
> provider (which requires Tomcat 8.5 or later).
>
>> <Connector port="8889" SSLCertificateFile="C:/mycer.cer"
>> SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true"
>> acceptCount="100" clientAuth="false" disableUploadTimeout="true"
>> enableLookups="false" keystoreFile="conf/.keystore"
>> maxHttpHeaderSize="8192" maxSavePostSize="-1" maxThreads="200"
>> minSpareThreads="20" protocol="HTTP/1.1" scheme="https"
>> secure="true" sslProtocol="TLS1.2"
>> compressableMimeTypes="text/html,text/xml,text/plain,application/json"
>>
>>
> compressionMinSize="2048"
>> compression="force" threadPriority="6" />
>
> When you say "high load", what kind of load are you talking about,
> specifically?
>
> You may simply have too much traffic for your existing hardware to
> handle. No amount of configuration can fix that.
> - -chris
We have 9 JVMs. As per current setting, per JVM max concurrency can
reach up-to 300(maxThreads="200"+acceptCount="100"). Issue was
reported when 2300 users on the system. So, I think it is not a case
of thread exhaustion but something else. Most possibly few JVMs were
under stress due to uneven load balancing(we have stickysession). This
issue just came to me and all resource usage data are gone. I have
asked for re-run so that I can analyze it properly. Let's see. :)
Back to my original question, in case of HTTP to HTTPS auto
redirection, Connector properties set for the later will be finally
applied to the request - is that a correct statement?
Thanks!
Suvendu
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Which Connector properties will be used in case of redirection?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Suvendu,
Which version of Tomcat?
On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote:
> I am seeing one issue. Under high load sporadically one web
> service call fails with: "java.net.SocketTimeoutException: Read
> timed out" after 60 Sec mark.
>
> In our app, httpd routes requests to 8888 then it get redirected
> to 8889.
Can you be very explicit about what you mean by "redirect", here?
> Connector which listens on 8888 has connectionTimeout=20000 but the
> one which listens to 8889 does not have any connectionTimeout set
> explicitly. As per doc, default connectionTimeout value 60 Sec will
> be set for connector which listens on 8889 - and that is playing a
> role here. Do you think this assumption is correct?
>
> Here are the connector properties: <Connector port="8888"
> protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8889"
> />
So you are probably expecting that all HTTP traffic to :8888 will be
redirected to HTTPS over :8889?
How long does the redirect take?
Under heavy load, you might be running out of threads depending upon a
lot of factors, such as what kind of protocol and endpoint are
actually being used. It looks like you are using the APR connector
(due to SSLCertificateFile) which has some positive attributes with
request to scalability while NIO would probably be the best choice if
possible.
It's got terrible performance, though, unless you use the OpenSSL
provider (which requires Tomcat 8.5 or later).
> <Connector port="8889" SSLCertificateFile="C:/mycer.cer"
> SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true"
> acceptCount="100" clientAuth="false" disableUploadTimeout="true"
> enableLookups="false" keystoreFile="conf/.keystore"
> maxHttpHeaderSize="8192" maxSavePostSize="-1" maxThreads="200"
> minSpareThreads="20" protocol="HTTP/1.1" scheme="https"
> secure="true" sslProtocol="TLS1.2"
> compressableMimeTypes="text/html,text/xml,text/plain,application/json"
>
>
compressionMinSize="2048"
> compression="force" threadPriority="6" />
When you say "high load", what kind of load are you talking about,
specifically?
You may simply have too much traffic for your existing hardware to
handle. No amount of configuration can fix that.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpfg7sdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFj12Q//WlJMd0JpQJVqFt1z
d3aI52pVFQih3Bii8UOAZzZr6omPqqFlkKt2cnAg8lWA3KQaQLJRyzlJKthIzNsB
12nMQlbQzCJZqn1IzggxtEWNL8KDVFY3fo2heJLrEANrIcXx+C6MAb1FLTeZ2/XP
V9R1u5q8mOdVgPyj4vITIsjn0oaa4pZ/ypkYwRATT9BRBy8rxkzVneDKPo8ILfdZ
Cp1Says1JD4uIEJn9VUlITOC1/08XyeHtrZXR3j3w9A07x9mLdHjVi6yxyyGcMoz
u0X/bt1E+Q6LkfTaQ64ubh2dbPXW4JgQy8I43Qzbxytwc2SdB691Un8lAhdnxaIk
zBioeiR4Kp+ATOuwbSPqJZ8n15pJENp95KvtR7B+xKn2klKMhuXHMb98OwdL7XBu
AeoGW1+RNpEwq1A8l4XmrmvFmrkqKTJVM8kxrPM3lTpTuQcvEtKk0OHOqlD10TGm
ldxP61qBLPK1ekjF/8CWxCO011wIvCoWySI8FDLOeXAo8F0ueBA0okajB2+9usqK
KpMckGQuRsoC6yVKMAz+ej3UCMiqQPSuuK06Pv8UrI7qTRGqRkBtVcSOIucXKC0S
6QrB4Mm6HajD7+d2/bOd4lJGRSSx3IoWNRx62Hthu9H2MTpp1/EWTwc5aesq2Zmo
OVsym/pIs8akxDw3Fkux5Imwr40=
=DLo7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org