You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Musomi Motilewa <mn...@philasd.org> on 2019/08/06 15:04:00 UTC
Cluster: Secondary Server's Farm War Deployer fails to move war to deploy directory on
Hi-
I'm currently running a two node cluster of Tomcat servers. I'm trying to implment Farm War Deployer (org.apache.catalina.ha.deploy.FarmWarDeployer) to deploy my war files to the cluster. The deployer is successfully deploying wars on the primary node when placed in the watch directory. On the secondary node the war file makes it to the tmp directory but is not deployed. The secondary node returns the following error:
06-Aug-2019 09:56:31.082 SEVERE [Tribes-Task-Receiver[localhost-Channel]-3] org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived Failed to rename [/var/tmp/war-tmp/ServerHealth.war] to [/opt/tomcat/latest/webapps/ServerHealth.war]
Server Configuration
- Apache Tomcat/9.0.22,
- openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment 19.3 (build 12.0.2+9)
OpenJDK 64-Bit Server VM 19.3 (build 12.0.2+9, mixed mode, sharing)
The <Cluster> tag is placed in the <Host> tag on both machines. The deployer tag is setup as follows with watchEnabled set to true on the primary server and false on the secondary. "{{ tomcat_McastAddress }}" is replaced by Ansible.
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="{{ tomcat_McastAddress }}"
port="45564"
frequency="500"
dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif|.*\.js|.*\.jpg|.*\.jpeg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt"/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/var/tmp/war-tmp/"
deployDir="/opt/tomcat/latest/webapps/"
watchDir="/var/tmp/war-watch/"
watchEnabled="{{ tomcat_deployerWatchEnabled|default('false', true) }}"
processDeployFrequency="2" />
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
The tempDir, deployDir and watchDir have the following owner and group permissions set.
/var/tmp/
drwxrwxr-x. 2 tomcat tomcat 30 Aug 6 09:56 war-tmp
drwxrwxr-x. 2 tomcat tomcat 6 Aug 6 08:18 war-watch
/opt/tomcat/lastest/
drwxr-x---. 7 tomcat tomcat 81 Aug 6 09:45 webapps
A checksum of the war file on both servers match so the file is being transfered to the secondary server's tempDir correctly. It just doesnt seem to be getting transered from the tempdir to the deployDir successfully.
Any help in this matter would be greatly appreciated. Thanks.
-Musomi
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Cluster: Secondary Server's Farm War Deployer fails to move war
to deploy directory on
Posted by "Motilewa, Musomi" <mn...@philasd.org>.
Chris-
Thanks for you help. The different file systems was the issue. I moved both the tempDir and watchDirs under /opt and all wars were deployed successfully. Thanks again for your quick response and help in this matter.
- Musomi
----- Original Message -----
From: "Christopher Schultz" <ch...@christopherschultz.net>
To: users@tomcat.apache.org
Sent: Tuesday, August 6, 2019 3:49:08 PM
Subject: Re: Cluster: Secondary Server's Farm War Deployer fails to move war to deploy directory on
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Musomi,
> Musomi,
>
> On 8/6/19 11:04, Musomi Motilewa wrote:
>> Hi-
>
>> I'm currently running a two node cluster of Tomcat servers. I'm
>> trying to implment Farm War Deployer
>> (org.apache.catalina.ha.deploy.FarmWarDeployer) to deploy my war
>> files to the cluster. The deployer is successfully deploying
>> wars on the primary node when placed in the watch directory. On
>> the secondary node the war file makes it to the tmp directory but
>> is not deployed. The secondary node returns the following error:
>
>> 06-Aug-2019 09:56:31.082 SEVERE
>> [Tribes-Task-Receiver[localhost-Channel]-3]
>> org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived
>> Failed to rename [/var/tmp/war-tmp/ServerHealth.war] to
>> [/opt/tomcat/latest/webapps/ServerHealth.war]
>
>> Server Configuration
>
>> - Apache Tomcat/9.0.22, - openjdk 12.0.2 2019-07-16 OpenJDK
>> Runtime Environment 19.3 (build 12.0.2+9) OpenJDK 64-Bit Server
>> VM 19.3 (build 12.0.2+9, mixed mode, sharing)
>
>> The <Cluster> tag is placed in the <Host> tag on both machines.
>> The deployer tag is setup as follows with watchEnabled set to
>> true on the primary server and false on the secondary. "{{
>> tomcat_McastAddress }}" is replaced by Ansible.
>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="8">
>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="{{ tomcat_McastAddress }}" port="45564" frequency="500"
>> dropTime="3000"/>
>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="auto" port="4000" autoBind="100" selectorTimeout="5000"
>> maxThreads="6"/>
>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitte
r
>
>>
">
>
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSend
er
>
>
"/>
>> </Sender>
>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDe
t
>
>>
ector"/>
>
>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispat
ch
>
>
Interceptor"/>
>> </Channel>
>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=".*\.gif|.*\.js|.*\.jpg|.*\.jpeg|.*\.png|.*\.htm|.*\.html|.*\.
c
>
>>
ss|.*\.txt"/>
>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
>> <Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/var/tmp/war-tmp/"
>> deployDir="/opt/tomcat/latest/webapps/"
>> watchDir="/var/tmp/war-watch/" watchEnabled="{{
>> tomcat_deployerWatchEnabled|default('false', true) }}"
>> processDeployFrequency="2" />
>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>
>>
>
> </Cluster>
>
>> The tempDir, deployDir and watchDir have the following owner and
>> group permissions set.
>
>> /var/tmp/ drwxrwxr-x. 2 tomcat tomcat 30 Aug 6 09:56 war-tmp
>> drwxrwxr-x. 2 tomcat tomcat 6 Aug 6 08:18 war-watch
>
>> /opt/tomcat/lastest/ drwxr-x---. 7 tomcat tomcat 81 Aug 6
>> 09:45 webapps
>
>> A checksum of the war file on both servers match so the file is
>> being transfered to the secondary server's tempDir correctly. It
>> just doesnt seem to be getting transered from the tempdir to the
>> deployDir successfully.
>
>> Any help in this matter would be greatly appreciated. Thanks.
>
> What do you get when you run these commands on the "secondary
> node"?
>
> $ id tomcat
>
> $ ps aux | grep catalina.base
>
> $ mount
>
> My guess is that your server is running as "tomcat" and your file
> permissions are correct. I'm guessing that /var/tmp and
> /opt/tomcat are on different filesystems and that File.rename
> doesn't work across filesystems in your environment.
Confirmed on Linux with ext4 and tmpfs, Java 1.8 File.renameTo will
return false and perform no action across those two filesystems.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1J2bQACgkQHPApP6U8
pFjcAw/8DdF1bIoPgEaf3wgP9HEbHLhRA6pHEcW5tQZA/RIE/dwmCGTq/6hLu7Q/
brbyEWz0df1RCngCW/Wdls/39E2BOuQAliLiYj3OdApCG4UuUKe6K90PofSxHCte
YhQcejq17jEtA9K359QV0bVfIhX3Kksziv1A7ymRC4YFd8NbZSruHbv1TmKQU9ws
IYZKxCGqoiqVymDN2EjRyOe/XeDo5RSxZixMBEhAlx8vg33Je/GFQy9xlR1Y3G0O
SEfFrGOM3jhUVePL+4kyEMaN/COTLqUjyTQn+mcA+jbTJfm7wWJlc3XqaxWT2tUr
AzuIqSLDz6SOwGKWotZB9yzlVLLreNA2GHcWcpb1aC5p79XtaZ2jhOX6tkZcNZmg
QZr4MEQVKM02A4Ls9zPCPVzygIbfpaaIhkaetM7E7oc5q4uCy8Cmso4stgAJ4Uc6
i964Dnn77Shn5p7H0AETWWJGgNG+S0FxP3gzaN4kxKevtzDjWCFmt5g6P39E/QyM
UQwvjKIUpLJs1tx+0ZhamCRfZy0N9KHypR8WbQUKZvQ3hUVOv9StvYJ0R5PENcjO
kqJtAWUAaDBV/Na4VhGTun/haDUcJ6p78+FRVr0gTiv3OGgYHSiJFxS04OWYzp+r
AwmMTGIDBt934B1osyMhvnrzET6pJsQigqTl48YnRz2D/b3Xkzs=
=4t+l
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Cluster: Secondary Server's Farm War Deployer fails to move war
to deploy directory on
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Musomi,
> Musomi,
>
> On 8/6/19 11:04, Musomi Motilewa wrote:
>> Hi-
>
>> I'm currently running a two node cluster of Tomcat servers. I'm
>> trying to implment Farm War Deployer
>> (org.apache.catalina.ha.deploy.FarmWarDeployer) to deploy my war
>> files to the cluster. The deployer is successfully deploying
>> wars on the primary node when placed in the watch directory. On
>> the secondary node the war file makes it to the tmp directory but
>> is not deployed. The secondary node returns the following error:
>
>> 06-Aug-2019 09:56:31.082 SEVERE
>> [Tribes-Task-Receiver[localhost-Channel]-3]
>> org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived
>> Failed to rename [/var/tmp/war-tmp/ServerHealth.war] to
>> [/opt/tomcat/latest/webapps/ServerHealth.war]
>
>> Server Configuration
>
>> - Apache Tomcat/9.0.22, - openjdk 12.0.2 2019-07-16 OpenJDK
>> Runtime Environment 19.3 (build 12.0.2+9) OpenJDK 64-Bit Server
>> VM 19.3 (build 12.0.2+9, mixed mode, sharing)
>
>> The <Cluster> tag is placed in the <Host> tag on both machines.
>> The deployer tag is setup as follows with watchEnabled set to
>> true on the primary server and false on the secondary. "{{
>> tomcat_McastAddress }}" is replaced by Ansible.
>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="8">
>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="{{ tomcat_McastAddress }}" port="45564" frequency="500"
>> dropTime="3000"/>
>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="auto" port="4000" autoBind="100" selectorTimeout="5000"
>> maxThreads="6"/>
>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitte
r
>
>>
">
>
>
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSend
er
>
>
"/>
>> </Sender>
>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDe
t
>
>>
ector"/>
>
>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispat
ch
>
>
Interceptor"/>
>> </Channel>
>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=".*\.gif|.*\.js|.*\.jpg|.*\.jpeg|.*\.png|.*\.htm|.*\.html|.*\.
c
>
>>
ss|.*\.txt"/>
>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
>> <Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/var/tmp/war-tmp/"
>> deployDir="/opt/tomcat/latest/webapps/"
>> watchDir="/var/tmp/war-watch/" watchEnabled="{{
>> tomcat_deployerWatchEnabled|default('false', true) }}"
>> processDeployFrequency="2" />
>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>
>>
>
> </Cluster>
>
>> The tempDir, deployDir and watchDir have the following owner and
>> group permissions set.
>
>> /var/tmp/ drwxrwxr-x. 2 tomcat tomcat 30 Aug 6 09:56 war-tmp
>> drwxrwxr-x. 2 tomcat tomcat 6 Aug 6 08:18 war-watch
>
>> /opt/tomcat/lastest/ drwxr-x---. 7 tomcat tomcat 81 Aug 6
>> 09:45 webapps
>
>> A checksum of the war file on both servers match so the file is
>> being transfered to the secondary server's tempDir correctly. It
>> just doesnt seem to be getting transered from the tempdir to the
>> deployDir successfully.
>
>> Any help in this matter would be greatly appreciated. Thanks.
>
> What do you get when you run these commands on the "secondary
> node"?
>
> $ id tomcat
>
> $ ps aux | grep catalina.base
>
> $ mount
>
> My guess is that your server is running as "tomcat" and your file
> permissions are correct. I'm guessing that /var/tmp and
> /opt/tomcat are on different filesystems and that File.rename
> doesn't work across filesystems in your environment.
Confirmed on Linux with ext4 and tmpfs, Java 1.8 File.renameTo will
return false and perform no action across those two filesystems.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1J2bQACgkQHPApP6U8
pFjcAw/8DdF1bIoPgEaf3wgP9HEbHLhRA6pHEcW5tQZA/RIE/dwmCGTq/6hLu7Q/
brbyEWz0df1RCngCW/Wdls/39E2BOuQAliLiYj3OdApCG4UuUKe6K90PofSxHCte
YhQcejq17jEtA9K359QV0bVfIhX3Kksziv1A7ymRC4YFd8NbZSruHbv1TmKQU9ws
IYZKxCGqoiqVymDN2EjRyOe/XeDo5RSxZixMBEhAlx8vg33Je/GFQy9xlR1Y3G0O
SEfFrGOM3jhUVePL+4kyEMaN/COTLqUjyTQn+mcA+jbTJfm7wWJlc3XqaxWT2tUr
AzuIqSLDz6SOwGKWotZB9yzlVLLreNA2GHcWcpb1aC5p79XtaZ2jhOX6tkZcNZmg
QZr4MEQVKM02A4Ls9zPCPVzygIbfpaaIhkaetM7E7oc5q4uCy8Cmso4stgAJ4Uc6
i964Dnn77Shn5p7H0AETWWJGgNG+S0FxP3gzaN4kxKevtzDjWCFmt5g6P39E/QyM
UQwvjKIUpLJs1tx+0ZhamCRfZy0N9KHypR8WbQUKZvQ3hUVOv9StvYJ0R5PENcjO
kqJtAWUAaDBV/Na4VhGTun/haDUcJ6p78+FRVr0gTiv3OGgYHSiJFxS04OWYzp+r
AwmMTGIDBt934B1osyMhvnrzET6pJsQigqTl48YnRz2D/b3Xkzs=
=4t+l
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Cluster: Secondary Server's Farm War Deployer fails to move war
to deploy directory on
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Musomi,
On 8/6/19 11:04, Musomi Motilewa wrote:
> Hi-
>
> I'm currently running a two node cluster of Tomcat servers. I'm
> trying to implment Farm War Deployer
> (org.apache.catalina.ha.deploy.FarmWarDeployer) to deploy my war
> files to the cluster. The deployer is successfully deploying wars
> on the primary node when placed in the watch directory. On the
> secondary node the war file makes it to the tmp directory but is
> not deployed. The secondary node returns the following error:
>
> 06-Aug-2019 09:56:31.082 SEVERE
> [Tribes-Task-Receiver[localhost-Channel]-3]
> org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived
> Failed to rename [/var/tmp/war-tmp/ServerHealth.war] to
> [/opt/tomcat/latest/webapps/ServerHealth.war]
>
> Server Configuration
>
> - Apache Tomcat/9.0.22, - openjdk 12.0.2 2019-07-16 OpenJDK Runtime
> Environment 19.3 (build 12.0.2+9) OpenJDK 64-Bit Server VM 19.3
> (build 12.0.2+9, mixed mode, sharing)
>
> The <Cluster> tag is placed in the <Host> tag on both machines. The
> deployer tag is setup as follows with watchEnabled set to true on
> the primary server and false on the secondary. "{{
> tomcat_McastAddress }}" is replaced by Ansible.
>
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="{{ tomcat_McastAddress }}" port="45564" frequency="500"
> dropTime="3000"/>
>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto" port="4000" autoBind="100" selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter
">
>
>
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender
"/>
> </Sender>
>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDet
ector"/>
>
>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch
Interceptor"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=".*\.gif|.*\.js|.*\.jpg|.*\.jpeg|.*\.png|.*\.htm|.*\.html|.*\.c
ss|.*\.txt"/>
>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/var/tmp/war-tmp/"
> deployDir="/opt/tomcat/latest/webapps/"
> watchDir="/var/tmp/war-watch/" watchEnabled="{{
> tomcat_deployerWatchEnabled|default('false', true) }}"
> processDeployFrequency="2" />
>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>
>
</Cluster>
>
> The tempDir, deployDir and watchDir have the following owner and
> group permissions set.
>
> /var/tmp/ drwxrwxr-x. 2 tomcat tomcat 30 Aug 6 09:56 war-tmp
> drwxrwxr-x. 2 tomcat tomcat 6 Aug 6 08:18 war-watch
>
> /opt/tomcat/lastest/ drwxr-x---. 7 tomcat tomcat 81 Aug 6 09:45
> webapps
>
> A checksum of the war file on both servers match so the file is
> being transfered to the secondary server's tempDir correctly. It
> just doesnt seem to be getting transered from the tempdir to the
> deployDir successfully.
>
> Any help in this matter would be greatly appreciated. Thanks.
What do you get when you run these commands on the "secondary node"?
$ id tomcat
$ ps aux | grep catalina.base
$ mount
My guess is that your server is running as "tomcat" and your file
permissions are correct. I'm guessing that /var/tmp and /opt/tomcat
are on different filesystems and that File.rename doesn't work across
filesystems in your environment.
The only ways around this problem would be to either change the watch
directory to be on the same filesystem or to patch the Tomcat code.
IMHO Tomcat should try harder to re-name the file by copying/removing
if the File.renameTo doesn't work. But you'd have to either patch it
locally and re-build or wait for another release of Tomcat in order to
get that running.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1J1UQACgkQHPApP6U8
pFiHxQ//faSXWXCXfRKfLpMFCw1VaMBcc9X9W+gyCWElkTJQkCQiq9LZEUDHXLo4
tNSgQAjHXshz5LyoGLouUueQsxyurlzLMkoUOnRPtEjWqpyGFx4DUXU4kf1jBs9F
UT0RwtspNzUH7KBEQmRJw3YpyXR3ld0yUrAndzZzzbzqYxx0Z9exNC/3AUKAa52F
KmVt8NXHDF4QG7mdEw5hFvl60vOh5aw8L9s/r1WTSXkl8tGEQ3j2esZMOZ1kh2aw
SrJk510WbtSLQ7Qx9iAC97wBYqA1FFzrNSrFnAlsDl7T98+xOxkF/n5BibGTk49j
dJWmGOX47kwCzLl0hBGs0rjtmwBPuIKC/3Dwstuuahp8HP+DP0niZQmopZgyCRXa
587AHZt6kzxXdnO2grssvy1Y3oNoXqcQuhM3B54ixvT2XeUoZpWfMFuhC8gaUDrg
NTJLRG5wviAqcPsHD9/mN9unhb/nAUW0AXO5EQVjhRL7bruofynI61pFNaQ601VY
/qjchGEskEOtHOOIrcl9/zYbzKpvIVqAKaRHhGMQtLg1iA8fVd+33zUcaGEeJV6V
4n2f4A6ctU8c/z9RV83SJ+biFJWCWC2Yot6MsI++x6bGgBmxRQWJ0auE+kZMmFnl
N3e/fo3FOnMTR2z2FoxEgZ2pOvSQm53vzBDRWO9/HO1fekVSYd0=
=GNUy
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org