You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomee.apache.org by Marco DE BOOIJ <ma...@debooy.eu> on 2020/07/21 09:43:27 UTC
TomEE on Docker
I had set-up TomEE (8.0.1) in Docker and Apache2 with mod_jk and it
worked. After a crash I lost all configurations. After re-installing
Linux (Mint 20) I re-created the TomEE 8.0.2 container (from tomitribe
on github with 11-jre but the 8-jre has the same problem) and configured
Apache2 (2.4.41) with the mod_jk (1.2.46-1).
I created the container (as before) with docker run -it --name tomee -p
8109:8009 -p 8180:8080 -p 8543:8443 -e TZ=Europe/Brussels -v
/srv/local/tomee:/srv/local/tomee tomee-8.0.2
When I access the applications through the TomEE port I get the reply
but when I go through Apache2 I get a "502 Bad Gateway The proxy server
received an invalid response from an upstream server". In the mod_jk.log
I find:
[Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13_worker) acquired
connection pool slot=0 after 0 retries
[Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13_worker) ajp
marshaling done
[Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
ajp_service::jk_ajp_common.c (2588): processing ajp13_worker with 2 retries
[Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
ajp_send_request::jk_ajp_common.c (1719): (ajp13_worker) no usable
connection found, will create a new one.
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
jk_open_socket::jk_connect.c (798): trying to connect socket 17 to
127.0.0.1:8109
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
jk_open_socket::jk_connect.c (824): socket 17 [127.0.0.1:44262 ->
127.0.0.1:8109] connected
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to
ajp13 pos=4 len=462 max=8192
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
ajp_send_request::jk_ajp_common.c (1779): (ajp13_worker) request body to
send 0 - request body to resend 0
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
jk_shutdown_socket::jk_connect.c (931): About to shutdown socket 17
[errno=107]
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
jk_shutdown_socket::jk_connect.c (940): Failed sending SHUT_WR for
socket 17 [errno=107]
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
ajp_connection_tcp_get_message::jk_ajp_common.c (1347): (ajp13_worker)
can't receive the response header message from tomcat, network problems
or tomcat (127.0.0.1:8109) is down (errno=104)
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
ajp_abort_endpoint::jk_ajp_common.c (818): (ajp13_worker) aborting
endpoint with socket 17
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [error]
ajp_get_reply::jk_ajp_common.c (2256): (ajp13_worker) Tomcat is down or
refused connection. No response has been sent to the client (yet)
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
ajp_service::jk_ajp_common.c (2775): (ajp13_worker) sending request to
tomcat failed (recoverable), (attempt=1)
[Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
ajp_service::jk_ajp_common.c (2624): (ajp13_worker) retry 1, sleeping
for 100 ms before retrying
I only commented out the AJP 1.3 connector part from the server.xml
without changing. I changed the port in workers.properties to 8109.
On the internet I found old problems but these should have been solved
in the version that I use. Did anybody had this problem and solved it?
Re: TomEE on Docker
Posted by Marco DE BOOIJ <ma...@debooy.eu>.
I did not change anything in the httpd-jk.conf file :
<IfModule jk_module>
# We need a workers file exactly once
# and in the global server
JkWorkersFile /etc/libapache2-mod-jk/workers.properties
# Our JK error log
# You can (and should) use rotatelogs here
JkLogFile /var/log/apache2/mod_jk.log
# Our JK log level (trace,debug,info,warn,error)
JkLogLevel info
# Our JK shared memory file
JkShmFile /var/log/apache2/jk-runtime-status
# Define a new log format you can use in any CustomLog in order
# to add mod_jk specific information to your access log.
# LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\" \"%{Cookie}i\" \"%{Set-Cookie}o\" %{pid}P %{tid}P
%{JK_LB_FIRST_NAME}n %{JK_LB_LAST_NAME}n ACC %{JK_LB_LAST_ACCESSED}n ERR
%{JK_LB_LAST_ERRORS}n BSY %{JK_LB_LAST_BUSY}n %{JK_LB_LAST_STATE}n %D"
extended_jk
# This option will reject all requests, which contain an
# encoded percent sign (%25) or backslash (%5C) in the URL
# If you are sure, that your webapp doesn't use such
# URLs, enable the option to prevent double encoding
attacks.httpd-jk.conf
# Since: 1.2.24
# JkOptions +RejectUnsafeURI
# After setting JkStripSession to "On", mod_jk will
# strip all ";jsessionid=..." from request URLs it
# does *not* forward to a backend.
# This is useful, if all links in a webapp use
# URLencoded session IDs and parts of the static
# content should be delivered directly by Apache.
# Of course you can also do it with mod_rewrite.
# Since: 1.2.21
# JkStripSession On
# Start a separate thread for internal tasks like
# idle connection probing, connection pool resizing
# and load value decay.
# Run these tasks every JkWatchdogInterval seconds.
# Since: 1.2.27
JkWatchdogInterval 60
# Configure access to jk-status and jk-manager
# If you want to make this available in a virtual host,
# either move this block into the virtual host
# or copy it logically there by including "JkMountCopy On"
# in the virtual host.
# Add an appropriate authentication method here!
<Location /jk-status>
# Inside Location we can omit the URL in JkMount
JkMount jk-status
Require ip 127.0.0.1
</Location>
<Location /jk-manager>
# Inside Location we can omit the URL in JkMount
JkMount jk-manager
Require ip 127.0.0.1
</Location>
</IfModule>
I must say that after the installation of the installation of mod_jk
(with apt-get install libapache2-mod-jk) the conf file (jk.conf) of the
module was not present in the mods_enabled directory. I created this
file as a copy of the httpd-jk.conf file. Perhaps something wrong in the
repository that is used bu Linux Mint 20? I will check on the secret.
On 2020-07-21 12:35, Jonathan Gallimore wrote:
> Hi
>
> Can you share the Apache HTTPD set with mod_jk config with us (I assume you
> run that in a separate Docker container)? Please don't share any sensitive
> config like passwords, keys, etc, just the basics of what we'd need to
> reproduce the error you're seeing. We'd be happy to take a look.
>
> There was a recent change in Tomcat where you need to set a secret both in
> the Tomcat/TomEE config, and on the mod_jk side. Checking that the AJP port
> is open and accessible to HTTPD, and that the secret is set on both sides
> would be my first step in troubleshooting this.
>
> The recent changes in AJP were to mitigate CVE-2020-1938 - some details are
> here:https://nvd.nist.gov/vuln/detail/CVE-2020-1938 and there are a number
> of writeups about the vulnerability on the web. Its worth a read and
> understanding the changes.
>
> Jon
>
> On Tue, Jul 21, 2020 at 10:43 AM Marco DE BOOIJ <ma...@debooy.eu>
> wrote:
>
>> I had set-up TomEE (8.0.1) in Docker and Apache2 with mod_jk and it
>> worked. After a crash I lost all configurations. After re-installing
>> Linux (Mint 20) I re-created the TomEE 8.0.2 container (from tomitribe
>> on github with 11-jre but the 8-jre has the same problem) and configured
>> Apache2 (2.4.41) with the mod_jk (1.2.46-1).
>>
>> I created the container (as before) with docker run -it --name tomee -p
>> 8109:8009 -p 8180:8080 -p 8543:8443 -e TZ=Europe/Brussels -v
>> /srv/local/tomee:/srv/local/tomee tomee-8.0.2
>>
>> When I access the applications through the TomEE port I get the reply
>> but when I go through Apache2 I get a "502 Bad Gateway The proxy server
>> received an invalid response from an upstream server". In the mod_jk.log
>> I find:
>>
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13_worker) acquired
>> connection pool slot=0 after 0 retries
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13_worker) ajp
>> marshaling done
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_service::jk_ajp_common.c (2588): processing ajp13_worker with 2 retries
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_send_request::jk_ajp_common.c (1719): (ajp13_worker) no usable
>> connection found, will create a new one.
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_open_socket::jk_connect.c (798): trying to connect socket 17 to
>> 127.0.0.1:8109
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_open_socket::jk_connect.c (824): socket 17 [127.0.0.1:44262 ->
>> 127.0.0.1:8109] connected
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to
>> ajp13 pos=4 len=462 max=8192
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_send_request::jk_ajp_common.c (1779): (ajp13_worker) request body to
>> send 0 - request body to resend 0
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_shutdown_socket::jk_connect.c (931): About to shutdown socket 17
>> [errno=107]
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_shutdown_socket::jk_connect.c (940): Failed sending SHUT_WR for
>> socket 17 [errno=107]
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
>> ajp_connection_tcp_get_message::jk_ajp_common.c (1347): (ajp13_worker)
>> can't receive the response header message from tomcat, network problems
>> or tomcat (127.0.0.1:8109) is down (errno=104)
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_abort_endpoint::jk_ajp_common.c (818): (ajp13_worker) aborting
>> endpoint with socket 17
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [error]
>> ajp_get_reply::jk_ajp_common.c (2256): (ajp13_worker) Tomcat is down or
>> refused connection. No response has been sent to the client (yet)
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
>> ajp_service::jk_ajp_common.c (2775): (ajp13_worker) sending request to
>> tomcat failed (recoverable), (attempt=1)
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_service::jk_ajp_common.c (2624): (ajp13_worker) retry 1, sleeping
>> for 100 ms before retrying
>>
>> I only commented out the AJP 1.3 connector part from the server.xml
>> without changing. I changed the port in workers.properties to 8109.
>>
>> On the internet I found old problems but these should have been solved
>> in the version that I use. Did anybody had this problem and solved it?
>>
>>
Re: TomEE on Docker
Posted by Marco DE BOOIJ <ma...@debooy.eu>.
Think I solved the problem. I added requiredSecret="aSecret" (not the
real security :-)) to the AJP/13 connector in the server.xml of TomEE
and worker.ajp13.secret=aSecret into the workers.properties file. After
the restart of both TomEE and Apache2 the problem was not gone. I found
another error message in the catalina.log: java.net.SocketException:
Protocol family unavailable. On the internet I found that you need to
change the ::1 in the AJP/13 connector in the server.xml into 0.0.0.0 to
switch to IPv4. Now after the restart of TomEE I was able to access my
applications through Apache. Is this the right way? What if I want to
limit the acces between the docker container and Apache2?
On 2020-07-21 12:35, Jonathan Gallimore wrote:
> Hi
>
> Can you share the Apache HTTPD set with mod_jk config with us (I assume you
> run that in a separate Docker container)? Please don't share any sensitive
> config like passwords, keys, etc, just the basics of what we'd need to
> reproduce the error you're seeing. We'd be happy to take a look.
>
> There was a recent change in Tomcat where you need to set a secret both in
> the Tomcat/TomEE config, and on the mod_jk side. Checking that the AJP port
> is open and accessible to HTTPD, and that the secret is set on both sides
> would be my first step in troubleshooting this.
>
> The recent changes in AJP were to mitigate CVE-2020-1938 - some details are
> here:https://nvd.nist.gov/vuln/detail/CVE-2020-1938 and there are a number
> of writeups about the vulnerability on the web. Its worth a read and
> understanding the changes.
>
> Jon
>
> On Tue, Jul 21, 2020 at 10:43 AM Marco DE BOOIJ <ma...@debooy.eu>
> wrote:
>
>> I had set-up TomEE (8.0.1) in Docker and Apache2 with mod_jk and it
>> worked. After a crash I lost all configurations. After re-installing
>> Linux (Mint 20) I re-created the TomEE 8.0.2 container (from tomitribe
>> on github with 11-jre but the 8-jre has the same problem) and configured
>> Apache2 (2.4.41) with the mod_jk (1.2.46-1).
>>
>> I created the container (as before) with docker run -it --name tomee -p
>> 8109:8009 -p 8180:8080 -p 8543:8443 -e TZ=Europe/Brussels -v
>> /srv/local/tomee:/srv/local/tomee tomee-8.0.2
>>
>> When I access the applications through the TomEE port I get the reply
>> but when I go through Apache2 I get a "502 Bad Gateway The proxy server
>> received an invalid response from an upstream server". In the mod_jk.log
>> I find:
>>
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13_worker) acquired
>> connection pool slot=0 after 0 retries
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13_worker) ajp
>> marshaling done
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_service::jk_ajp_common.c (2588): processing ajp13_worker with 2 retries
>> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
>> ajp_send_request::jk_ajp_common.c (1719): (ajp13_worker) no usable
>> connection found, will create a new one.
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_open_socket::jk_connect.c (798): trying to connect socket 17 to
>> 127.0.0.1:8109
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_open_socket::jk_connect.c (824): socket 17 [127.0.0.1:44262 ->
>> 127.0.0.1:8109] connected
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to
>> ajp13 pos=4 len=462 max=8192
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_send_request::jk_ajp_common.c (1779): (ajp13_worker) request body to
>> send 0 - request body to resend 0
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_shutdown_socket::jk_connect.c (931): About to shutdown socket 17
>> [errno=107]
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> jk_shutdown_socket::jk_connect.c (940): Failed sending SHUT_WR for
>> socket 17 [errno=107]
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
>> ajp_connection_tcp_get_message::jk_ajp_common.c (1347): (ajp13_worker)
>> can't receive the response header message from tomcat, network problems
>> or tomcat (127.0.0.1:8109) is down (errno=104)
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_abort_endpoint::jk_ajp_common.c (818): (ajp13_worker) aborting
>> endpoint with socket 17
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [error]
>> ajp_get_reply::jk_ajp_common.c (2256): (ajp13_worker) Tomcat is down or
>> refused connection. No response has been sent to the client (yet)
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
>> ajp_service::jk_ajp_common.c (2775): (ajp13_worker) sending request to
>> tomcat failed (recoverable), (attempt=1)
>> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
>> ajp_service::jk_ajp_common.c (2624): (ajp13_worker) retry 1, sleeping
>> for 100 ms before retrying
>>
>> I only commented out the AJP 1.3 connector part from the server.xml
>> without changing. I changed the port in workers.properties to 8109.
>>
>> On the internet I found old problems but these should have been solved
>> in the version that I use. Did anybody had this problem and solved it?
>>
>>
Re: TomEE on Docker
Posted by Jonathan Gallimore <jo...@gmail.com>.
Hi
Can you share the Apache HTTPD set with mod_jk config with us (I assume you
run that in a separate Docker container)? Please don't share any sensitive
config like passwords, keys, etc, just the basics of what we'd need to
reproduce the error you're seeing. We'd be happy to take a look.
There was a recent change in Tomcat where you need to set a secret both in
the Tomcat/TomEE config, and on the mod_jk side. Checking that the AJP port
is open and accessible to HTTPD, and that the secret is set on both sides
would be my first step in troubleshooting this.
The recent changes in AJP were to mitigate CVE-2020-1938 - some details are
here:https://nvd.nist.gov/vuln/detail/CVE-2020-1938 and there are a number
of writeups about the vulnerability on the web. Its worth a read and
understanding the changes.
Jon
On Tue, Jul 21, 2020 at 10:43 AM Marco DE BOOIJ <ma...@debooy.eu>
wrote:
> I had set-up TomEE (8.0.1) in Docker and Apache2 with mod_jk and it
> worked. After a crash I lost all configurations. After re-installing
> Linux (Mint 20) I re-created the TomEE 8.0.2 container (from tomitribe
> on github with 11-jre but the 8-jre has the same problem) and configured
> Apache2 (2.4.41) with the mod_jk (1.2.46-1).
>
> I created the container (as before) with docker run -it --name tomee -p
> 8109:8009 -p 8180:8080 -p 8543:8443 -e TZ=Europe/Brussels -v
> /srv/local/tomee:/srv/local/tomee tomee-8.0.2
>
> When I access the applications through the TomEE port I get the reply
> but when I go through Apache2 I get a "502 Bad Gateway The proxy server
> received an invalid response from an upstream server". In the mod_jk.log
> I find:
>
> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
> ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13_worker) acquired
> connection pool slot=0 after 0 retries
> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
> ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13_worker) ajp
> marshaling done
> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
> ajp_service::jk_ajp_common.c (2588): processing ajp13_worker with 2 retries
> [Fri Jul 17 11:09:27.272 2020] [1075:140607154546432] [debug]
> ajp_send_request::jk_ajp_common.c (1719): (ajp13_worker) no usable
> connection found, will create a new one.
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> jk_open_socket::jk_connect.c (798): trying to connect socket 17 to
> 127.0.0.1:8109
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> jk_open_socket::jk_connect.c (824): socket 17 [127.0.0.1:44262 ->
> 127.0.0.1:8109] connected
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to
> ajp13 pos=4 len=462 max=8192
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> ajp_send_request::jk_ajp_common.c (1779): (ajp13_worker) request body to
> send 0 - request body to resend 0
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> jk_shutdown_socket::jk_connect.c (931): About to shutdown socket 17
> [errno=107]
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> jk_shutdown_socket::jk_connect.c (940): Failed sending SHUT_WR for
> socket 17 [errno=107]
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
> ajp_connection_tcp_get_message::jk_ajp_common.c (1347): (ajp13_worker)
> can't receive the response header message from tomcat, network problems
> or tomcat (127.0.0.1:8109) is down (errno=104)
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> ajp_abort_endpoint::jk_ajp_common.c (818): (ajp13_worker) aborting
> endpoint with socket 17
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [error]
> ajp_get_reply::jk_ajp_common.c (2256): (ajp13_worker) Tomcat is down or
> refused connection. No response has been sent to the client (yet)
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [info]
> ajp_service::jk_ajp_common.c (2775): (ajp13_worker) sending request to
> tomcat failed (recoverable), (attempt=1)
> [Fri Jul 17 11:09:27.273 2020] [1075:140607154546432] [debug]
> ajp_service::jk_ajp_common.c (2624): (ajp13_worker) retry 1, sleeping
> for 100 ms before retrying
>
> I only commented out the AJP 1.3 connector part from the server.xml
> without changing. I changed the port in workers.properties to 8109.
>
> On the internet I found old problems but these should have been solved
> in the version that I use. Did anybody had this problem and solved it?
>
>