You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Tomcat Random <to...@gmail.com> on 2013/09/19 17:39:20 UTC
ARP Connector questions
Tomcat 7.0.42, RHEL6
I've installed the ARP connector and have my service.xml configured with
the only enabled connector being:
<!--APR connector native installed -->
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" scheme="http" secure="false"
SSLEnabled="false"/>
1. I'm expecting about 2500 simultaneous visitors. Any thoughts on how much
I might want to bump up the maxThreads and acceptCount?
2. Does the ARP connector work within the Executor thread pools? I'm a
little unclear on this. Currently the Executor node is commented out. Do I
want a shared executor for the ARP connector?
Thanks in advance,
Alec
Re: APR Connector questions
Posted by Tomcat Random <to...@gmail.com>.
Ok, thanks for the advice. If it means removing one more layer of
complexity, I'm all for it.
Best,
Alec
On Fri, Sep 20, 2013 at 11:57 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Alec,
>
> On 9/20/13 2:03 PM, Tomcat Random wrote:
> > Chris, Thanks for correcting the misdirected reply.
> >
> > "Do you mean it's not working under load, or you haven't yet tried
> > it under load?"
> >
> > I mean I haven't tested it under load yet.
> >
> > One of the main reasons for choosing APR is is that I was under
> > the impression it's good at serving large static files. My site has
> > some large swf files - one is ~8mb (it's a game site).
> >
> > The docs on APR say, "When APR is enabled, the HTTP connector will
> > use sendfile for handling large static files (all such files will
> > be sent asynchronously using high performance kernel level calls),
> > and will use a socket poller for keepalive, increasing scalability
> > of the server."
>
> The NIO connector also supports sendfile, and avoids blocking I/O for
> keepalives.
>
> > Your recommendation is NIO is comparable to APR in non-ssl
> > performance (as above for large static files) and more stable, and
> > it would be better to switch to that?
>
> I'm suggesting that NIO might be a comparable choice to APR with the
> added benefit of removing some complexity (that of installing a native
> library) to your deployment.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJSPRk7AAoJEBzwKT+lPKRYv5oP/1JZhzlnNfhPI7LLPJKegf5T
> Nk1Hd5DDPLoa58ihhnntQ0le0dOK6x+6LcktoIr1qvP9q2IiBupwF2HRnW4okW9O
> Db3p4vr0/9IioRPCkHpEcG6J+JR0L93SEDucqFpLQCAaR6x9Yc/ziGqO241sPhJD
> 5BmEBPBi+Kl+OD+UNhrMpyzNKf/zdmzjJu7oMl97DS6kNmx6gf2rvEwBS2Iec6xV
> NgfzqQ/6faSIsFv5AseHIXmYkZcifyegUYemQt+ZtNs7z9C0rx7Gd1Hh6ls2mjlG
> WD4Y2yILg8WouDZXJXEhGU5Pq65iVCoYPTWTF4tvJS0aU4AYVx5opiSZNeZy6vGl
> UAsX7lpTDgQ/VXfEOHmslvZsHorkOnh6z9CcVDtjcZYf+mFouGy3CXJROTcUizJg
> pzwghiT4jX9xcUWaf13CjuqBMo5SwsSqkkf4HY2vFDBDfn70bIG8k+FdjjTjKjv1
> hZwkGc4Ysc0h0b2vKCYgI78fwydDvdnoNEJ50IONP6coxo4fSdaFCaFCQ/gXKVLG
> puMVkbE5WAkgxFBcM0zms5U9oqAQ2ZnwlGMB6tM1/GvnIQYgAiqqDVgEwm/wbWct
> XYxPIHakMXtJZRPY5lECQzmbHMZX4HnJ/si53lKQ2JeT79JC+Pesox0fNobU2eD1
> K5Wu5Y96NL5F+Frl3wOE
> =9/4N
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: APR Connector questions
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Alec,
On 9/20/13 2:03 PM, Tomcat Random wrote:
> Chris, Thanks for correcting the misdirected reply.
>
> "Do you mean it's not working under load, or you haven't yet tried
> it under load?"
>
> I mean I haven't tested it under load yet.
>
> One of the main reasons for choosing APR is is that I was under
> the impression it's good at serving large static files. My site has
> some large swf files - one is ~8mb (it's a game site).
>
> The docs on APR say, "When APR is enabled, the HTTP connector will
> use sendfile for handling large static files (all such files will
> be sent asynchronously using high performance kernel level calls),
> and will use a socket poller for keepalive, increasing scalability
> of the server."
The NIO connector also supports sendfile, and avoids blocking I/O for
keepalives.
> Your recommendation is NIO is comparable to APR in non-ssl
> performance (as above for large static files) and more stable, and
> it would be better to switch to that?
I'm suggesting that NIO might be a comparable choice to APR with the
added benefit of removing some complexity (that of installing a native
library) to your deployment.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJSPRk7AAoJEBzwKT+lPKRYv5oP/1JZhzlnNfhPI7LLPJKegf5T
Nk1Hd5DDPLoa58ihhnntQ0le0dOK6x+6LcktoIr1qvP9q2IiBupwF2HRnW4okW9O
Db3p4vr0/9IioRPCkHpEcG6J+JR0L93SEDucqFpLQCAaR6x9Yc/ziGqO241sPhJD
5BmEBPBi+Kl+OD+UNhrMpyzNKf/zdmzjJu7oMl97DS6kNmx6gf2rvEwBS2Iec6xV
NgfzqQ/6faSIsFv5AseHIXmYkZcifyegUYemQt+ZtNs7z9C0rx7Gd1Hh6ls2mjlG
WD4Y2yILg8WouDZXJXEhGU5Pq65iVCoYPTWTF4tvJS0aU4AYVx5opiSZNeZy6vGl
UAsX7lpTDgQ/VXfEOHmslvZsHorkOnh6z9CcVDtjcZYf+mFouGy3CXJROTcUizJg
pzwghiT4jX9xcUWaf13CjuqBMo5SwsSqkkf4HY2vFDBDfn70bIG8k+FdjjTjKjv1
hZwkGc4Ysc0h0b2vKCYgI78fwydDvdnoNEJ50IONP6coxo4fSdaFCaFCQ/gXKVLG
puMVkbE5WAkgxFBcM0zms5U9oqAQ2ZnwlGMB6tM1/GvnIQYgAiqqDVgEwm/wbWct
XYxPIHakMXtJZRPY5lECQzmbHMZX4HnJ/si53lKQ2JeT79JC+Pesox0fNobU2eD1
K5Wu5Y96NL5F+Frl3wOE
=9/4N
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: APR Connector questions
Posted by Tomcat Random <to...@gmail.com>.
Chris, Thanks for correcting the misdirected reply.
"Do you mean it's not working under load, or you haven't yet tried it
under load?"
I mean I haven't tested it under load yet.
One of the main reasons for choosing APR is is that I was under the
impression it's good at serving large static files. My site has some large
swf files - one is ~8mb (it's a game site).
The docs on APR say, "When APR is enabled, the HTTP connector will use
sendfile for handling large static files (all such files will be sent
asynchronously using high performance kernel level calls), and will use a
socket poller for keepalive, increasing scalability of the server."
Your recommendation is NIO is comparable to APR in non-ssl performance (as
above for large static files) and more stable, and it would be better to
switch to that?
Thanks again,
Alec
On Thu, Sep 19, 2013 at 1:56 PM, Jeffrey Janner <Jeffrey.Janner@polydyne.com
> wrote:
> > -----Original Message-----
> > From: Christopher Schultz [mailto:chris@christopherschultz.net]
> > Sent: Thursday, September 19, 2013 12:38 PM
> > To: Tomcat Users List
> > Cc: Tomcat Random
> > Subject: Re: APR Connector questions
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > Alec,
> >
> > Please keep discussions on the mailing lists so others can benefit from
> > them.
> >
> > On 9/19/13 12:01 PM, Tomcat Random wrote:
> > > The answer for am I going to be using SSL is maybe. It's not
> > > mandatory, but would be nice for an admin area of the site. I already
> > > built the tcnative stuff and APR is working, but not under load.
> >
> > Do you mean it's not working under load, or you haven't yet tried it
> > under load?
> >
> > > I'm using APR because it was my understanding there was a big
> > > performance increase when using Tomcat without a proxy/web server in
> > > front of it. I just have Tomcat with my IP tables redirecting
> > > 80->8080.
> >
> > APR and NIO performance are comparably to each other, SSL excepted. If
> > you are talking about using SSL only for "admin" access (which is
> > usually fairly limited in scope/traffic), then I wouldn't worry about
> > the performance difference.
> >
> > One could argue that any site that requires login should be 100% SSL-
> > protected, but I know nothing about your requirements.
> >
>
> +1
>
> > > "2500 users might not require 2500- simultaneous connections."
> > > True, and it occurs to me, sort of noobishly, where would you look
> > for
> > > reporting simultaneous connections?
> >
> > You can use JMX to get lots of information about the connectors.
> > You'll have to probe periodically and build-up a trend graph to
> > understand your actual traffic.
> >
> > http://wiki.apache.org/tomcat/FAQ/Monitoring
> >
> > > And once you know that number, back to my original question, how many
> > > maxthreads/acceptCounts?
> >
> > The acceptCount is just the TCP backlog. Setting this higher than the
> > default is only helpful if you have huge transaction volume bursts and
> > your transactions are fairly short. If you can't handle 200
> > transactions waiting in the TCP accept queue pretty quickly, it's not
> > going to help to raise that number to 1000. If you experience huge
> > bursts of traffic that your app can handle with a short delay -- AND if
> > you absolutely don't want to give any clients "connection refused"
> > errors -- then raising the acceptCount is appropriate. I haven't seen a
> > "normal" webapp that has ever required changing from the default, but
> > my experience may not match the type of business you are in.
> >
> > As for maxThreads, that depends upon your load, the type of hardware
> > you have, the length of your transactions, and the CPU load you expect
> > will be required for your webapp. If your webapp is fairly CPU-bound
> > (which I've found to be fairly rare) and you have a limited number of
> > physical CPUs, raising the maxThreads limit buys you nothing: it may be
> > worse than lowering it because you just end up running many threads at
> > once and thrash the CPU.
> >
> > If you have a primarily I/O-bound app (most that I've seen... e.g.
> > stuff that uses back-end databases for most requests) than raising the
> > maxThreads can serve more requests... but then remember that your
> > database must be able to handle the load as well. Having 1000 worker
> > threads with a DB connection pool of size=10 means lots of waiting
> > threads.
> >
> > > Just how rare are the APR catastrophes?
> >
> > I don't have much data on frequency of occurrences.... just what I can
> > see in BZ for the Tomcat Connectors project.
> >
>
> Let's just say that over the past 8 or so years, I've yet to have it
> happen to me, and I am supporting dozens of Tomcat instances across a
> half-dozen systems with each Tomcat having 5 or 6 hosts each. Then again,
> I'm running under Windows and the tcnative is built for me by the good guys
> on the Tomcat Dev Team.
>
> > > Is it something a tomcat restart can fix?
> >
> > You don't have a choice: the JVM goes down immediately and you *must*
> > restart Tomcat. That's what I meant by "catastrophic".
> >
>
> Yes, unfortunately, anything that causes a crash at the native code level
> is going to stop everything.
> A restart may not "fix" the problem, but you can usually at least recover
> to a normal state for a time. At least until whatever specific
> circumstances that caused the crash occur again.
>
> Jeff
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: APR Connector questions
Posted by Jeffrey Janner <Je...@PolyDyne.com>.
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: Thursday, September 19, 2013 12:38 PM
> To: Tomcat Users List
> Cc: Tomcat Random
> Subject: Re: APR Connector questions
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Alec,
>
> Please keep discussions on the mailing lists so others can benefit from
> them.
>
> On 9/19/13 12:01 PM, Tomcat Random wrote:
> > The answer for am I going to be using SSL is maybe. It's not
> > mandatory, but would be nice for an admin area of the site. I already
> > built the tcnative stuff and APR is working, but not under load.
>
> Do you mean it's not working under load, or you haven't yet tried it
> under load?
>
> > I'm using APR because it was my understanding there was a big
> > performance increase when using Tomcat without a proxy/web server in
> > front of it. I just have Tomcat with my IP tables redirecting
> > 80->8080.
>
> APR and NIO performance are comparably to each other, SSL excepted. If
> you are talking about using SSL only for "admin" access (which is
> usually fairly limited in scope/traffic), then I wouldn't worry about
> the performance difference.
>
> One could argue that any site that requires login should be 100% SSL-
> protected, but I know nothing about your requirements.
>
+1
> > "2500 users might not require 2500- simultaneous connections."
> > True, and it occurs to me, sort of noobishly, where would you look
> for
> > reporting simultaneous connections?
>
> You can use JMX to get lots of information about the connectors.
> You'll have to probe periodically and build-up a trend graph to
> understand your actual traffic.
>
> http://wiki.apache.org/tomcat/FAQ/Monitoring
>
> > And once you know that number, back to my original question, how many
> > maxthreads/acceptCounts?
>
> The acceptCount is just the TCP backlog. Setting this higher than the
> default is only helpful if you have huge transaction volume bursts and
> your transactions are fairly short. If you can't handle 200
> transactions waiting in the TCP accept queue pretty quickly, it's not
> going to help to raise that number to 1000. If you experience huge
> bursts of traffic that your app can handle with a short delay -- AND if
> you absolutely don't want to give any clients "connection refused"
> errors -- then raising the acceptCount is appropriate. I haven't seen a
> "normal" webapp that has ever required changing from the default, but
> my experience may not match the type of business you are in.
>
> As for maxThreads, that depends upon your load, the type of hardware
> you have, the length of your transactions, and the CPU load you expect
> will be required for your webapp. If your webapp is fairly CPU-bound
> (which I've found to be fairly rare) and you have a limited number of
> physical CPUs, raising the maxThreads limit buys you nothing: it may be
> worse than lowering it because you just end up running many threads at
> once and thrash the CPU.
>
> If you have a primarily I/O-bound app (most that I've seen... e.g.
> stuff that uses back-end databases for most requests) than raising the
> maxThreads can serve more requests... but then remember that your
> database must be able to handle the load as well. Having 1000 worker
> threads with a DB connection pool of size=10 means lots of waiting
> threads.
>
> > Just how rare are the APR catastrophes?
>
> I don't have much data on frequency of occurrences.... just what I can
> see in BZ for the Tomcat Connectors project.
>
Let's just say that over the past 8 or so years, I've yet to have it happen to me, and I am supporting dozens of Tomcat instances across a half-dozen systems with each Tomcat having 5 or 6 hosts each. Then again, I'm running under Windows and the tcnative is built for me by the good guys on the Tomcat Dev Team.
> > Is it something a tomcat restart can fix?
>
> You don't have a choice: the JVM goes down immediately and you *must*
> restart Tomcat. That's what I meant by "catastrophic".
>
Yes, unfortunately, anything that causes a crash at the native code level is going to stop everything.
A restart may not "fix" the problem, but you can usually at least recover to a normal state for a time. At least until whatever specific circumstances that caused the crash occur again.
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: APR Connector questions
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Alec,
Please keep discussions on the mailing lists so others can benefit
from them.
On 9/19/13 12:01 PM, Tomcat Random wrote:
> The answer for am I going to be using SSL is maybe. It's not
> mandatory, but would be nice for an admin area of the site. I
> already built the tcnative stuff and APR is working, but not under
> load.
Do you mean it's not working under load, or you haven't yet tried it
under load?
> I'm using APR because it was my understanding there was a big
> performance increase when using Tomcat without a proxy/web server
> in front of it. I just have Tomcat with my IP tables redirecting
> 80->8080.
APR and NIO performance are comparably to each other, SSL excepted. If
you are talking about using SSL only for "admin" access (which is
usually fairly limited in scope/traffic), then I wouldn't worry about
the performance difference.
One could argue that any site that requires login should be 100%
SSL-protected, but I know nothing about your requirements.
> "2500 users might not require 2500- simultaneous connections."
> True, and it occurs to me, sort of noobishly, where would you look
> for reporting simultaneous connections?
You can use JMX to get lots of information about the connectors.
You'll have to probe periodically and build-up a trend graph to
understand your actual traffic.
http://wiki.apache.org/tomcat/FAQ/Monitoring
> And once you know that number, back to my original question, how
> many maxthreads/acceptCounts?
The acceptCount is just the TCP backlog. Setting this higher than the
default is only helpful if you have huge transaction volume bursts and
your transactions are fairly short. If you can't handle 200
transactions waiting in the TCP accept queue pretty quickly, it's not
going to help to raise that number to 1000. If you experience huge
bursts of traffic that your app can handle with a short delay -- AND
if you absolutely don't want to give any clients "connection refused"
errors -- then raising the acceptCount is appropriate. I haven't seen
a "normal" webapp that has ever required changing from the default,
but my experience may not match the type of business you are in.
As for maxThreads, that depends upon your load, the type of hardware
you have, the length of your transactions, and the CPU load you expect
will be required for your webapp. If your webapp is fairly CPU-bound
(which I've found to be fairly rare) and you have a limited number of
physical CPUs, raising the maxThreads limit buys you nothing: it may
be worse than lowering it because you just end up running many threads
at once and thrash the CPU.
If you have a primarily I/O-bound app (most that I've seen... e.g.
stuff that uses back-end databases for most requests) than raising the
maxThreads can serve more requests... but then remember that your
database must be able to handle the load as well. Having 1000 worker
threads with a DB connection pool of size=10 means lots of waiting
threads.
> Just how rare are the APR catastrophes?
I don't have much data on frequency of occurrences.... just what I can
see in BZ for the Tomcat Connectors project.
> Is it something a tomcat restart can fix?
You don't have a choice: the JVM goes down immediately and you *must*
restart Tomcat. That's what I meant by "catastrophic".
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJSOzZ/AAoJEBzwKT+lPKRYL+0QAKMQIlr2crCQ35nbop8BvSGl
GpxfNiVhmNdVLwzNIp68xoQQhpYHMcg/ukFLjwaGG6AORlxf39PLFizd0TMKJNtO
EK2ZCRq6X4WXIwTPIaCb0ovbVp3HxIXPs+VxPVaDcxiHSCQN0FoaQFgP35E91rDR
ymgATcj+XXbc8rH5DyRMSPExbzO/5SeBWM65uQLfa2iA/qLCFq8NsrZgvi6QxZjR
O9JX+hWogL1nTPmgyyyXqY1YtIzTpB7F35itCCcccAkbmt4YePb7joQ4pPPX2jvT
aAggfHg9YUWoUuRA4IwCRRBkZiWSc8vkQhwV6vBN7Bw7/pK7x0SmwsBtBeZFkYa2
kxP8BOAVbsZu7ahnqGVIY17ul8FF4lslfRWN2YY4qqNdShUXkcTMiOoQcCU8Y4FL
zPgmIlqUQ2KRP8F9+9/66RCMCv+RS4bxo6Aq+IeEoz0B7peWVLDMLORNc5D6Y2s8
P/0ImtczB/wvEPdKpRVqB2uuDJfOBUD0vFGv3/TH8WXHIJyIiwK+Up3vtFlAEJjm
DLgO7W18mmEum81hrfvt4JXFFCV9kqHhVpFHXXDcARL4qEwp4fI2DaEitj1e5fGk
FIIA/EIx3gU0vjy3yjqccO9WjvHkbotgrJuWgrdPz25VA5ceGqqUaSWxKP4mEooy
nZAhl1oF5Glv3rIyNkjN
=7erg
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: APR Connector questions (was: ARP Connector questions)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Alec,
Changed subject: it's APR (Apache Portable Runtime), not ARP (which is
something different).
On 9/19/13 11:39 AM, Tomcat Random wrote:
> Tomcat 7.0.42, RHEL6
>
> I've installed the APR connector and have my service.xml configured
> with the only enabled connector being:
>
> <!--APR connector native installed --> <Connector port="8080"
> maxHttpHeaderSize="8192" maxThreads="150" enableLookups="false"
> disableUploadTimeout="true" acceptCount="100" scheme="http"
> secure="false" SSLEnabled="false"/>
>
> 1. I'm expecting about 2500 simultaneous visitors. Any thoughts on
> how much I might want to bump up the maxThreads and acceptCount?
The better question is how many simultaneous /requests/ you expect.
2500 users might not require 2500- simultaneous connections.
> 2. Does the APR connector work within the Executor thread pools?
> I'm a little unclear on this. Currently the Executor node is
> commented out. Do I want a shared executor for the ARP connector?
Yes, you can use an executor. If you don't specify one, a <Connector>
will create a default <Executor> for itself. If you expect to have
multiple connectors and you want them all to share a pool of worker
threads, then you'll want to configure a single <Executor> and then
reference that from each of your <Connector>s.
Are you going to be using SSL? If not, you might have slightly better
luck with the NIO connector (APR/OpenSSL has a performance advantage
over JSSE), plus you won't have to build and babysit tcnative, etc.
Problems that occur at the NIO level will likely throw exceptions
while APR can bring-down the whole JVM. It's rare, but when it
happens, it's catastrophic.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJSOxwzAAoJEBzwKT+lPKRY2M8QAMqvN5KpOzmhBJbSlzQ770tr
xP8WiIrtloormFI9+XpT6ZJ2c1v5fx/MjlhSjhXKQd6M54pC9MKIXmM4zrpaOtPh
6amiRh7dNJXnYsUXKylw5O8lI4frsxbCnBS5wtXL1zywXL7uXbj37w4GbEzaYkql
5hL7z65/pBIhk9QTCfeUE7gvk1MQNMhkfDgGEHanWmwH3rCBucNjF7O8aUn8OlZy
3guTpvfo2TpPiSA6tpEgTliqR7ZTg19KN8flXtiN1+M/L+OGips1ihOPRYxj4Osc
A51pltqdEyv1cSR3oVyb4xYXCYHjO0kxoDNIpDr+HZp4Xw7bdb+VvXZamyffvOyP
dgs8zRRGsmKpVaPXnKcjdhNlbtCGKppQFuMeYxz975/dVxpHXcr3xmPKg33H31Jd
OnrYvyTjxeP6Y/cj1AYUZYwr+yzg3FBPv8S/O2POy1eZDTPVwr0uKfIlNYl6bzrr
PiEizoq/UYuIK+wst9NZsztAIf70VWmCbN1lW5oz090tXR0yU2xxdFhh3P7yUV/j
LxvQqGshX5uHW1sySGeznluof9t/RPd1938Sx0ML94EHmi+U7Mb+Y7u+jxy4skxT
m58Q3vxTYZrv78ayEfXP7KTdDXGItNwVwbILMpLopQ6oJe0N7ay8bbO5tj29/aR4
oOuFKKtVOXjWpZ+nC1Om
=AjzT
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org