You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by ji...@apache.org on 2017/07/18 21:00:14 UTC
svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Author: jim
Date: Tue Jul 18 21:00:14 2017
New Revision: 1802336
URL: http://svn.apache.org/viewvc?rev=1802336&view=rev
Log:
UDS does support reuse
Modified:
httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Modified: httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml?rev=1802336&r1=1802335&r2=1802336&view=diff
==============================================================================
--- httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml (original)
+++ httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml Tue Jul 18 21:00:14 2017
@@ -98,7 +98,6 @@ ProxyPassMatch "^/myapp/.*\.php(/.*)?$"
the hostname and optional port following fcgi:// are ignored.</p>
<example><title>PHP-FPM with UDS</title>
<highlight language="config">
-# UDS does not currently support connection reuse
ProxyPassMatch "^/(.*\.php(/.*)?)$" "unix:/var/run/php5-fpm.sock|fcgi://localhost/var/www/"
</highlight>
</example>
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jacob Champion <ch...@gmail.com>.
On 07/24/2017 11:47 AM, Helmut K. C. Tessarek wrote:
> I can try to reproduce it when I have set up a VM to run tests.
> Unfortunately I currently do not have the time to do so. Maybe in a
> few days.
Thanks! If this is the root cause of lost POSTs, then that'd be a good
reason to implement FCGI_GET_VALUES support in mod_proxy_fcgi so we can
query the maximum number of connections from the application itself.
--Jacob
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by "Helmut K. C. Tessarek" <te...@evermeet.cx>.
On 2017-07-24 14:38, Jacob Champion wrote:
> We can't really help debug your problems very well if we can't reproduce
> them. Knowing whether this is the root cause or not is important, so we
> can decide what to do next.
Oh, sorry, no I meant the bigger picture, not this specific problem.
I can try to reproduce it when I have set up a VM to run tests.
Unfortunately I currently do not have the time to do so.
Maybe in a few days.
In production I just disabled the entire Proxy section and everything
works now.
--
regards Helmut K. C. Tessarek KeyID 0xF7832007C11F128D
Key fingerprint = 28A3 1666 4FE8 D72C CFD5 8B23 F783 2007 C11F 128D
/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jacob Champion <ch...@gmail.com>.
vOn 07/24/2017 11:34 AM, Helmut K. C. Tessarek wrote:
> On 2017-07-24 13:42, Jacob Champion wrote:
>> Any chance this is because of the hang I mentioned?
>
> It could be, but this doesn't really change anything.
We can't really help debug your problems very well if we can't reproduce
them. Knowing whether this is the root cause or not is important, so we
can decide what to do next.
--Jacob
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by "Helmut K. C. Tessarek" <te...@evermeet.cx>.
On 2017-07-24 13:42, Jacob Champion wrote:
> Any chance this is because of the hang I mentioned?
It could be, but this doesn't really change anything.
If the setting enablereuse can change the behaviour of web applications
and introduces hangs and timeouts, it is time to rethink the
implementation. Maybe there should be a way to communicate between fcgi
backed and proxy, so that the request is not lost but sent on a previous
established connection.
This setting should improve performance, not make it possible to have
failed requests due to timeouts and/or hangs.
--
regards Helmut K. C. Tessarek KeyID 0xF7832007C11F128D
Key fingerprint = 28A3 1666 4FE8 D72C CFD5 8B23 F783 2007 C11F 128D
/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jacob Champion <ch...@gmail.com>.
On 07/24/2017 10:40 AM, Helmut K. C. Tessarek wrote:
> In my case it screwed up the POST request by not doing what should have
> been done with the request. In the end the request was lost.
> (due to timeout, not processing it, or for whatever reason)
Any chance this is because of the hang I mentioned?
--Jacob
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by "Helmut K. C. Tessarek" <te...@evermeet.cx>.
On 2017-07-24 13:29, Jacob Champion wrote:
> I have still not been able to reproduce the "messed up POSTs" that other
> people are reporting with UDS+enablereuse.
In my case it screwed up the POST request by not doing what should have
been done with the request. In the end the request was lost.
(due to timeout, not processing it, or for whatever reason)
It did not mess up the content of the request, if that's what you're
thinking.
The request just did not get through. After removing the Proxy section
with the enablereuse option, all was good again.
--
regards Helmut K. C. Tessarek KeyID 0xF7832007C11F128D
Key fingerprint = 28A3 1666 4FE8 D72C CFD5 8B23 F783 2007 C11F 128D
/*
Thou shalt not follow the NULL pointer for chaos and madness
await thee at its end.
*/
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Jul 24, 2017 at 10:44 PM, Jacob Champion <ch...@gmail.com> wrote:
> On 07/24/2017 01:03 PM, Yann Ylavic wrote:
>>
>> What if the fpm queue in ap_proxy_connect_backend or at POLLOUT time?
>> If both cases I think the socket is not reused by mod_proxy_fcgi for
>> the next request, so I don't see how it's a enablereuse issue...
>
>
> I'm not sure I understand your question, but I think I've described the
> problem incompletely. Let me give it another try:
Thanks, I better see your point now.
>
> Let's say FPM has been configured to handle 5 concurrent connections,
> and httpd has been configured to use up to 10. With connection reuse
> disabled, if we hit the server with ten requests to FPM, five
> connections will be served immediately, and FPM will queue up the next
> five. As requests finish, httpd closes each connection, and FPM grabs
> the next connection to service. Everything works.
Right, until connections start to be queued on the httpd side too, we
won't service more quicky than the backend anyway.
>
> With connection reuse enabled, FPM still queues the last five
> connections. But because httpd never releases the first five, and new
> incoming requests continue to use them, those queued connections never
> get serviced.
I agree that httpd could be better about this, but mainly in the
documentation IMHO (sizing...), because like in the disablereuse case
if there were no more new requests than the capacity of servicing the
existing ones (within the configured timeout), everything would work
here too (with less fairness, granted).
> That's not FPM's fault; it advertises how many concurrent
> connections it can handle via the FCGI protocol, and if it refused those
> queued connections by default, our first use case (which works fine)
> would crash and burn.
OK, fpm behaves correctly (although it could maybe service more fairly
new vs kept alive connections, or even the other way around).
But httpd is not really at fault either, I think the issue is mainly
"administrative".
Regards,
Yann.
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jacob Champion <ch...@gmail.com>.
On 07/24/2017 01:03 PM, Yann Ylavic wrote:
> What if the fpm queue in ap_proxy_connect_backend or at POLLOUT time?
> If both cases I think the socket is not reused by mod_proxy_fcgi for
> the next request, so I don't see how it's a enablereuse issue...
I'm not sure I understand your question, but I think I've described the
problem incompletely. Let me give it another try:
Let's say FPM has been configured to handle 5 concurrent connections,
and httpd has been configured to use up to 10. With connection reuse
disabled, if we hit the server with ten requests to FPM, five
connections will be served immediately, and FPM will queue up the next
five. As requests finish, httpd closes each connection, and FPM grabs
the next connection to service. Everything works.
With connection reuse enabled, FPM still queues the last five
connections. But because httpd never releases the first five, and new
incoming requests continue to use them, those queued connections never
get serviced. That's not FPM's fault; it advertises how many concurrent
connections it can handle via the FCGI protocol, and if it refused those
queued connections by default, our first use case (which works fine)
would crash and burn.
--Jacob
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Jul 24, 2017 at 9:46 PM, Jacob Champion <ch...@gmail.com> wrote:
> On 07/24/2017 12:39 PM, Yann Ylavic wrote:
>>
>> Hmm, don't we close the backend connection (i.e. conn->close = 1) whenever
>> an error occurs in the fgci loop? What do you mean by
>> "queue up for later", by whom? Where do that coonection go on the
>> httpd side?
>
>
> I mean that the FCGI application (PHP-FPM) has a listen queue for
> connections. I haven't looked into the source to see how this queue is
> implemented.
What if the fpm queue in ap_proxy_connect_backend or at POLLOUT time?
If both cases I think the socket is not reused by mod_proxy_fcgi for
the next request, so I don't see how it's a enablereuse issue...
>
> FPM has a status page that can be set up to debug these sorts of things,
> which I might try to enable later this week for more research.
Thanks, hopefully we'll see more clearly what happens for such
connections on the httpd side.
> I think
> the "correct" way to go is to query the app for the max connection
> number, use that if the admin hasn't set up another max, and warn if the
> admin has forced us to use a max that's more than what the app can
> handle concurrently.
Sure the warning is nice to have, but having max < ThreadsPerChild is
really not ideal, so the true fix is sizing fpm's max connection to
httpd's max workers (rather than waiting for available connections on
the reverse proxy side).
In any case we should behave correctly (i.e. according to the
configured timeouts) when the backend queues connections.
Regards,
Yann.
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jacob Champion <ch...@gmail.com>.
On 07/24/2017 12:39 PM, Yann Ylavic wrote:
> Hmm, don't we close the backend connection (i.e. conn->close = 1)
> whenever an error occurs in the fgci loop? What do you mean by
> "queue up for later", by whom? Where do that coonection go on the
> httpd side?
I mean that the FCGI application (PHP-FPM) has a listen queue for
connections. I haven't looked into the source to see how this queue is
implemented.
FPM has a status page that can be set up to debug these sorts of things,
which I might try to enable later this week for more research. I think
the "correct" way to go is to query the app for the max connection
number, use that if the admin hasn't set up another max, and warn if the
admin has forced us to use a max that's more than what the app can
handle concurrently.
--Jacob
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Jul 24, 2017 at 7:29 PM, Jacob Champion <ch...@gmail.com> wrote:
>
> I was able to reproduce a hang with PHP-FPM and enablereuse=on, but I
> don't think it's UDS-specific. (Haven't tried with TCP yet, though.)
>
> If the maximum number of connections opened by httpd to PHP-FPM exceeds
> the maximum number of requests that FPM is configured to handle in
> parallel, the extra connections will be queued up for later. But since
> we're no longer closing connections once a request finishes, those
> queued connections will hang until we hit the ProxyTimeout.
Hmm, don't we close the backend connection (i.e. conn->close = 1)
whenever an error occurs in the fgci loop? What do you mean by "queue
up for later", by whom? Where do that coonection go on the httpd side?
A first look it seems we do close on error, which should take care of
not reusing a backend connection is an undefined state.
At reuse time (next request), we also check for connection reusability
with a nonblocking read, otherwise we open a new one.
Admittedly it won't address the timeout case of the previous request
(which can't/shouldn't be handled here anyway), but if the connection
was closed remotely in the meantime it would be detected, unless the
connection is really aside nowhere on the backend side...
Regards,
Yann.
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jacob Champion <ch...@gmail.com>.
On 07/19/2017 04:56 AM, Yann Ylavic wrote:
> The comment in proxy_fcgi says:
>
> /* This scheme handler does not reuse connections by default, to
> * avoid tying up a fastcgi that isn't expecting to work on
> * parallel requests. But if the user went out of their way to
> * type the default value of disablereuse=off, we'll allow it.
> */
>
> So could it be an fpm/application bug, maybe also suggested in the
> link provided by Luca where it's also said that the issue happens with
> other servers too (e.g. nginx)?
I was able to reproduce a hang with PHP-FPM and enablereuse=on, but I
don't think it's UDS-specific. (Haven't tried with TCP yet, though.)
If the maximum number of connections opened by httpd to PHP-FPM exceeds
the maximum number of requests that FPM is configured to handle in
parallel, the extra connections will be queued up for later. But since
we're no longer closing connections once a request finishes, those
queued connections will hang until we hit the ProxyTimeout. It's
important for admins to match the Proxy's max= parameter with their
backend's configuration, which is something you didn't have to worry
about as much with enablereuse=off.
I have still not been able to reproduce the "messed up POSTs" that other
people are reporting with UDS+enablereuse.
--Jacob
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Jul 19, 2017, at 12:44 PM, Luca Toscano <to...@gmail.com> wrote:
>
> What does reuse=on do more other than forcing a worker to reuse the same socket?
>
Reuse also activates the IP address cache...
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Eric Covener <co...@gmail.com>.
On Wed, Jul 19, 2017 at 12:44 PM, Luca Toscano <to...@gmail.com> wrote:
> Hi Yann,
>
> 2017-07-19 13:56 GMT+02:00 Yann Ylavic <yl...@gmail.com>:
>>
>> On Wed, Jul 19, 2017 at 1:45 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>> > There's nothing in the way we handle UDS that prevents "reuse"
>> > even understanding that we use the term "reuse" to mean both
>> > "cache the IP address" and "use the connection as a one-shot".
>> > There's no diff between UDS and sockets from the module PoV.
>>
>> +1
>
>
> +1 as well, this is my understanding too.
>
>>
>>
>> The comment in proxy_fcgi says:
>>
>> /* This scheme handler does not reuse connections by default, to
>> * avoid tying up a fastcgi that isn't expecting to work on
>> * parallel requests. But if the user went out of their way to
>> * type the default value of disablereuse=off, we'll allow it.
>>
>> */
>
>
> This comment in the code was also a bit weird, what does it mean "a fastcgi
> that isn't expecting to work on parallel requests" ? Even if reuse is off
> multiple FCGI requests can land to the same FCGI backend at the same time
> no? What does reuse=on do more other than forcing a worker to reuse the same
> socket?
I think this comment is OK, but the related comment in the manual is bad.
I think the original concern was some FastCGI that would not even
accept() two connections. Keeping one
idle in httpd process A would not allow another httpd process B to
dispatch to the FastCGI because it is waiting on the first connection.
But, this must be the case in only the most primitive hello-world of FastCGI's.
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Luca Toscano <to...@gmail.com>.
Hi Yann,
2017-07-19 13:56 GMT+02:00 Yann Ylavic <yl...@gmail.com>:
> On Wed, Jul 19, 2017 at 1:45 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> > There's nothing in the way we handle UDS that prevents "reuse"
> > even understanding that we use the term "reuse" to mean both
> > "cache the IP address" and "use the connection as a one-shot".
> > There's no diff between UDS and sockets from the module PoV.
>
> +1
>
+1 as well, this is my understanding too.
>
> The comment in proxy_fcgi says:
>
> /* This scheme handler does not reuse connections by default, to
> * avoid tying up a fastcgi that isn't expecting to work on
> * parallel requests. But if the user went out of their way to
> * type the default value of disablereuse=off, we'll allow it.
*/
>
This comment in the code was also a bit weird, what does it mean "a fastcgi
that isn't expecting to work on parallel requests" ? Even if reuse is off
multiple FCGI requests can land to the same FCGI backend at the same time
no? What does reuse=on do more other than forcing a worker to reuse the
same socket?
>
> So could it be an fpm/application bug, maybe also suggested in the
> link provided by Luca where it's also said that the issue happens with
> other servers too (e.g. nginx)?
>
I left a comment in the website asking for how to reproduce the issue, hope
to get an answer. If so I'll report back my findings :)
Luca
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jul 19, 2017 at 1:45 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> There's nothing in the way we handle UDS that prevents "reuse"
> even understanding that we use the term "reuse" to mean both
> "cache the IP address" and "use the connection as a one-shot".
> There's no diff between UDS and sockets from the module PoV.
+1
The comment in proxy_fcgi says:
/* This scheme handler does not reuse connections by default, to
* avoid tying up a fastcgi that isn't expecting to work on
* parallel requests. But if the user went out of their way to
* type the default value of disablereuse=off, we'll allow it.
*/
So could it be an fpm/application bug, maybe also suggested in the
link provided by Luca where it's also said that the issue happens with
other servers too (e.g. nginx)?
Regards,
Yann.
Re: svn commit: r1802336 -
/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Jim Jagielski <ji...@jaguNET.com>.
There's nothing in the way we handle UDS that prevents "reuse"
even understanding that we use the term "reuse" to mean both
"cache the IP address" and "use the connection as a one-shot".
There's no diff between UDS and sockets from the module PoV.
> On Jul 19, 2017, at 4:26 AM, Luca Toscano <to...@gmail.com> wrote:
>
> Hi Jim,
>
> 2017-07-18 23:00 GMT+02:00 <ji...@apache.org>:
> Author: jim
> Date: Tue Jul 18 21:00:14 2017
> New Revision: 1802336
>
> URL: http://svn.apache.org/viewvc?rev=1802336&view=rev
> Log:
> UDS does support reuse
>
> Modified:
> httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
>
> Modified: httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml?rev=1802336&r1=1802335&r2=1802336&view=diff
> ==============================================================================
> --- httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml (original)
> +++ httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml Tue Jul 18 21:00:14 2017
> @@ -98,7 +98,6 @@ ProxyPassMatch "^/myapp/.*\.php(/.*)?$"
> the hostname and optional port following fcgi:// are ignored.</p>
> <example><title>PHP-FPM with UDS</title>
> <highlight language="config">
> -# UDS does not currently support connection reuse
> ProxyPassMatch "^/(.*\.php(/.*)?)$" "unix:/var/run/php5-fpm.sock|fcgi://localhost/var/www/"
> </highlight>
> </example>
>
>
> Thanks for the clarification, I was trying to figure out why we had that warning in the docs but didn't find anything useful. While chatting with Daniel (ezra-s) on IRC it came up that some users reported on #httpd that mod_proxy_fcgi errors out while re-using UDS sockets, but he didn't recall the specific issue. In a recent dev@ thread this problem came up again, so I did a little research and the best report that I found is the following:
>
> http://z-issue.com/wp/apache-2-4-the-event-mpm-php-via-mod_proxy_fcgi-and-php-fpm-with-vhosts/#comment-6765
>
> The user found the "UDS does not currently support connection reuse" comment and solved the issue, so I am wondering if this was a misconfiguration from his side or a known php-fpm limitation or a bug in mod_proxy_fcgi. I haven't been able to reproduce and this might totally be a false alarm, but if anybody has more info please speak up so we can investigate it properly :)
>
> Luca
Re: svn commit: r1802336 - /httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
Posted by Luca Toscano <to...@gmail.com>.
Hi Jim,
2017-07-18 23:00 GMT+02:00 <ji...@apache.org>:
> Author: jim
> Date: Tue Jul 18 21:00:14 2017
> New Revision: 1802336
>
> URL: http://svn.apache.org/viewvc?rev=1802336&view=rev
> Log:
> UDS does support reuse
>
> Modified:
> httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
>
> Modified: httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/m
> od/mod_proxy_fcgi.xml?rev=1802336&r1=1802335&r2=1802336&view=diff
> ============================================================
> ==================
> --- httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml (original)
> +++ httpd/httpd/trunk/docs/manual/mod/mod_proxy_fcgi.xml Tue Jul 18
> 21:00:14 2017
> @@ -98,7 +98,6 @@ ProxyPassMatch "^/myapp/.*\.php(/.*)?$"
> the hostname and optional port following fcgi:// are ignored.</p>
> <example><title>PHP-FPM with UDS</title>
> <highlight language="config">
> -# UDS does not currently support connection reuse
> ProxyPassMatch "^/(.*\.php(/.*)?)$" "unix:/var/run/php5-fpm.sock|f
> cgi://localhost/var/www/"
> </highlight>
> </example>
>
Thanks for the clarification, I was trying to figure out why we had that
warning in the docs but didn't find anything useful. While chatting with
Daniel (ezra-s) on IRC it came up that some users reported on #httpd that
mod_proxy_fcgi errors out while re-using UDS sockets, but he didn't recall
the specific issue. In a recent dev@ thread this problem came up again, so
I did a little research and the best report that I found is the following:
http://z-issue.com/wp/apache-2-4-the-event-mpm-php-via-mod_p
roxy_fcgi-and-php-fpm-with-vhosts/#comment-6765
The user found the "UDS does not currently support connection reuse"
comment and solved the issue, so I am wondering if this was a
misconfiguration from his side or a known php-fpm limitation or a bug in
mod_proxy_fcgi. I haven't been able to reproduce and this might totally be
a false alarm, but if anybody has more info please speak up so we can
investigate it properly :)
Luca