You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2021/08/25 19:57:21 UTC
[Bug 65519] New: mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
Bug ID: 65519
Summary: mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Product: Apache httpd-2
Version: 2.4.48
Hardware: PC
OS: Linux
Status: NEW
Severity: regression
Priority: P2
Component: mod_proxy_http
Assignee: bugs@httpd.apache.org
Reporter: mariustant@yahoo.com
Target Milestone: ---
Created attachment 37988
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37988&action=edit
Test harness + logs
When httpd is used as a reverse proxy to WildFly, it sometimes hangs when a
Java client tries to connect to WildFly via jboss-remoting (EJB calls). This
works fine in httpd version 2.4.46.
This is an intermittent issue (in my testing, more than 50% of the connections
hang).
How to reproduce:
1. Deploy the attached EAR file (simple-ear.ear) in WildFly 13. The EAR
contains a single EJB that will be accessed by the client.
2. Configure httpd as a reverse proxy to WildFly (httpd should handle HTTP
protocol upgrade to jboss-remoting):
ProxyPass "/" "ws://localhost:1026/" upgrade=jboss-remoting
A sample configuration file is attached (httpd.conf)
3. Run the attached standalone Java client that attempts to invoke an EJB in
WildFly via httpd. Before running the client, you need to copy the WildFly
client library (jboss-client.jar) to the lib folder (this is not included in
the archive due to its size).
cd echo-client-wf13
java -cp ./bin:./lib/jboss-client.jar:./lib/simple-ejbClient.jar
com.microfocus.echoclient.EchoClient "remote+https" "localhost" "4433"
Port 4433 is SSL enabled. For clear text communication use protocol as
"remote+http" and port as "80".
The client requires Java 8 or above (testing was done with Java 8).
When it works, the client displays the following:
Aug 25, 2021 2:54:46 PM org.wildfly.security.Version <clinit>
INFO: ELY00001: WildFly Elytron version 1.3.3.Final
Aug 25, 2021 2:54:46 PM org.wildfly.naming.client.Version <clinit>
INFO: WildFly Naming version 1.0.9.Final
Aug 25, 2021 2:54:47 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.6.3.Final
Aug 25, 2021 2:54:47 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.6.3.Final
Aug 25, 2021 2:54:47 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.2.Final
Aug 25, 2021 2:54:47 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 5.0.7.Final
Aug 25, 2021 2:54:47 PM org.jboss.ejb.client.EJBClient <clinit>
INFO: JBoss EJB Client version 4.0.10.Final
Result from EchoService: hello
When it doesn't work, the client hangs and the last line from above is not
displayed.
The httpd error logs (LogLevel trace8 + dumpio) for the case when the call
hangs are in: error_log_blocked.txt
The last log entry in this case is:
Wed Aug 25 19:25:54.299051 2021] [proxy:trace8] [pid 31599] proxy_util.c(4800):
[client 127.0.0.1:58402] proxy: HTTP: polling (client=31, origin=31)
tcpdump on the WildFly port shows that after the protocol upgrade is accepted
by WildFly (status code 101), the jboss-remoting conversation begins by WildFly
sending another TCP packet to httpd.
This packet is acknowledged by the httpd socket but the data is not visible in
httpd (the polling does not return). See tcpdump1026-blocked.txt
The httpd error logs for the case when the call works are in: error_log_ok.txt
The corresponding tcpdump is in tcpdump1026-ok.txt
The issue does not reproduce when the following directive is added to httpd
conf: "ProxyWebsocketFallbackToProxyHttp Off". So it seems that when the
protocol upgrade is handled by mod_proxy_wstunnel everything is working as
expected.
Also, the issue only reproduces with WildFly 13. The same scenario executed
against WildFly 20 does not reproduce the issue. However this seems like a race
condition so it might reproduce even with WildFly 20 granted that more testing
is performed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #2 from Yann Ylavic <yl...@gmail.com> ---
Also on which platform is httpd running, Linux?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #10 from Marius Tantareanu <ma...@yahoo.com> ---
With the patch in place, I am not able to reproduce the issue anymore. Thanks
for the quick turnaround.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #11 from Yann Ylavic <yl...@gmail.com> ---
Fixed in 2.4.49 (r1892971).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #3 from Marius Tantareanu <ma...@yahoo.com> ---
(In reply to Yann Ylavic from comment #1)
> (In reply to Marius Tantareanu from comment #0)
> > Wed Aug 25 19:25:54.299051 2021] [proxy:trace8] [pid 31599]
> > proxy_util.c(4800): [client 127.0.0.1:58402] proxy: HTTP: polling
> > (client=31, origin=31)
>
> This call never returns? No timeout at some point?
The call times out after 60 minutes. But I need to check if the timeout is
triggered by httpd or by the WildFly client. I will try to reproduce and send
the logs for a timed out call.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #5 from Yann Ylavic <yl...@gmail.com> ---
Created attachment 38003
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38003&action=edit
Update data_in_input_filters in the core and ssl input filters
The "101 Switching Protocols" response header and the first following
jboss-remoting data may be read at the same time by httpd, leading to the
application data being buffered in the filters chain and the tunneling loop not
noticing.
Can you try this patch which should address this potential issue?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #8 from Marius Tantareanu <ma...@yahoo.com> ---
Created attachment 38005
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38005&action=edit
error_log capturing the pooling timeout after 60 minutes
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #7 from Marius Tantareanu <ma...@yahoo.com> ---
(In reply to Marius Tantareanu from comment #3)
> (In reply to Yann Ylavic from comment #1)
> > (In reply to Marius Tantareanu from comment #0)
> > > Wed Aug 25 19:25:54.299051 2021] [proxy:trace8] [pid 31599]
> > > proxy_util.c(4800): [client 127.0.0.1:58402] proxy: HTTP: polling
> > > (client=31, origin=31)
> >
> > This call never returns? No timeout at some point?
>
> The call times out after 60 minutes. But I need to check if the timeout is
> triggered by httpd or by the WildFly client. I will try to reproduce and
> send the logs for a timed out call.
It seems that the 60 minutes timeout is generated by httpd. I am attaching the
error_log for this case (error_log_timeout).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #9 from Marius Tantareanu <ma...@yahoo.com> ---
(In reply to Yann Ylavic from comment #6)
> Created attachment 38004 [details]
> Issue a nonblocking instead of ap_filter_input_pending()
>
> Actually a simpler/better patch might be this one, please try it instead.
Thanks! I will give it a try and get back to you.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #4 from Marius Tantareanu <ma...@yahoo.com> ---
(In reply to Yann Ylavic from comment #2)
> Also on which platform is httpd running, Linux?
Yes, Linux. I tested on the following platforms:
Red Hat Enterprise Linux Server release 7.5 (Maipo)
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 2
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #38003|0 |1
is obsolete| |
--- Comment #6 from Yann Ylavic <yl...@gmail.com> ---
Created attachment 38004
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38004&action=edit
Issue a nonblocking instead of ap_filter_input_pending()
Actually a simpler/better patch might be this one, please try it instead.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 65519] mod_proxy_http hangs after HTTP upgrade to
jboss-remoting
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=65519
--- Comment #1 from Yann Ylavic <yl...@gmail.com> ---
(In reply to Marius Tantareanu from comment #0)
> Wed Aug 25 19:25:54.299051 2021] [proxy:trace8] [pid 31599]
> proxy_util.c(4800): [client 127.0.0.1:58402] proxy: HTTP: polling
> (client=31, origin=31)
This call never returns? No timeout at some point?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org