You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by rp...@apache.org on 2005/12/11 01:15:33 UTC

svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Author: rpluem
Date: Sat Dec 10 16:15:27 2005
New Revision: 355823

URL: http://svn.apache.org/viewcvs?rev=355823&view=rev
Log:
* Move handling of backends that broke after the headers have been sent
  into the proxy handler of mod_proxy.

  This patch still sets r->connection->aborted to 1 which is currently
  vetoed by Roy. Moving it from the scheme handler to the proxy handler
  should ease the reimplementation of this, as the scheme handlers only
  needs to return PROXY_BACKEND_BROKEN to signal the above situation to
  the proxy handler.

  mod_proxy.h: Add define for PROXY_BACKEND_BROKEN
  mod_proxy.c: Handle PROXY_BACKEND_BROKEN in proxy handler
  mod_proxy_http.c: Sent back PROXY_BACKEND_BROKEN if backend broke
  after we sent the headers.

Modified:
    httpd/httpd/trunk/modules/proxy/mod_proxy.c
    httpd/httpd/trunk/modules/proxy/mod_proxy.h
    httpd/httpd/trunk/modules/proxy/mod_proxy_http.c

Modified: httpd/httpd/trunk/modules/proxy/mod_proxy.c
URL: http://svn.apache.org/viewcvs/httpd/httpd/trunk/modules/proxy/mod_proxy.c?rev=355823&r1=355822&r2=355823&view=diff
==============================================================================
--- httpd/httpd/trunk/modules/proxy/mod_proxy.c (original)
+++ httpd/httpd/trunk/modules/proxy/mod_proxy.c Sat Dec 10 16:15:27 2005
@@ -758,6 +758,20 @@
                 worker->s->status |= PROXY_WORKER_IN_ERROR;
             }
         }
+        else if (access_status == PROXY_BACKEND_BROKEN) {
+            /*
+             * If the backend broke after the headers had been sent do not
+             * try another worker, but leave. Do not mark the worker as
+             * unsuable as this problem may not reoccur on the next request.
+             *
+             * TODO: Currently we abort the connection and notify all parties
+             * on the upstream that something went wrong by setting c->aborted
+             * to 1. This idea is currently vetoed and should be replaced with
+             * other methods
+             */
+            r->connection->aborted = 1;
+            break;
+        }
         else {
             /* Unrecoverable error.
              * Return the origin status code to the client.

Modified: httpd/httpd/trunk/modules/proxy/mod_proxy.h
URL: http://svn.apache.org/viewcvs/httpd/httpd/trunk/modules/proxy/mod_proxy.h?rev=355823&r1=355822&r2=355823&view=diff
==============================================================================
--- httpd/httpd/trunk/modules/proxy/mod_proxy.h (original)
+++ httpd/httpd/trunk/modules/proxy/mod_proxy.h Sat Dec 10 16:15:27 2005
@@ -238,6 +238,12 @@
     proxy_conn_rec *conn;   /* Single connection for prefork mpm's */
 };
 
+/*
+ * Return code that scheme handlers should return if the backend connection
+ * broke after they have sent the headers
+ */
+#define PROXY_BACKEND_BROKEN -10
+
 /* woker status flags */
 #define PROXY_WORKER_INITIALIZED    0x0001
 #define PROXY_WORKER_IGNORE_ERRORS  0x0002

Modified: httpd/httpd/trunk/modules/proxy/mod_proxy_http.c
URL: http://svn.apache.org/viewcvs/httpd/httpd/trunk/modules/proxy/mod_proxy_http.c?rev=355823&r1=355822&r2=355823&view=diff
==============================================================================
--- httpd/httpd/trunk/modules/proxy/mod_proxy_http.c (original)
+++ httpd/httpd/trunk/modules/proxy/mod_proxy_http.c Sat Dec 10 16:15:27 2005
@@ -1199,6 +1199,7 @@
                            * are being read. */
     int pread_len = 0;
     apr_table_t *save_table;
+    int backend_broken = 0;
 
     bb = apr_brigade_create(p, c->bucket_alloc);
 
@@ -1486,7 +1487,7 @@
                          */
                         ap_log_cerror(APLOG_MARK, APLOG_ERR, rv, c,
                                       "proxy: error reading response");
-                        c->aborted = 1;
+                        backend_broken = 1;
                         break;
                     }
                     /* next time try a non-blocking read */
@@ -1552,9 +1553,9 @@
         }
     } while (interim_response);
 
-    /* If our connection with the client is to be aborted, return DONE. */
-    if (c->aborted) {
-        return DONE;
+    /* Signal back that the backend broke after we sent the headers. */
+    if (backend_broken) {
+        return PROXY_BACKEND_BROKEN;
     }
 
     if (conf->error_override) {



Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Sat, Dec 10, 2005 at 05:06:42PM -0800, Roy Fielding wrote:
> On Dec 10, 2005, at 5:02 PM, Justin Erenkrantz wrote:
> >That will get mod_cache not to pick up on it, right.  However, it  
> >won't
> >force the connection to close.  Or, give anything a chance to close  
> >it: be
> >it HTTP/1.1 or Waka.  =)  -- justin
> 
> Right, that was just the proxy part -- the other half is not done yet.

Then, yes, +1 to that part.  -- justin

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Dec 10, 2005, at 5:02 PM, Justin Erenkrantz wrote:
> That will get mod_cache not to pick up on it, right.  However, it  
> won't
> force the connection to close.  Or, give anything a chance to close  
> it: be
> it HTTP/1.1 or Waka.  =)  -- justin

Right, that was just the proxy part -- the other half is not done yet.

....Roy


Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 02:02 AM, Justin Erenkrantz wrote:
> On Sat, Dec 10, 2005 at 04:51:22PM -0800, Roy Fielding wrote:
> 

>>As it turns out, we also have a request_rec->no_cache variable that
>>is supposed to be set in this case.  I am curious whether the following
>>patch will work (after reverting r355823).

Reverted in the meantime.

> 
> 
> That will get mod_cache not to pick up on it, right.  However, it won't
> force the connection to close.  Or, give anything a chance to close it: be
> it HTTP/1.1 or Waka.  =)  -- justin

What if we set c->keepalive to AP_CONN_CLOSE?
I think mod_disk_cache should do the right thing with Roy's patch. The only
thing that bugs me is that this check is done in mod_disk_cache and not
in mod_cache. But this is independant from Roy's patch.
I guess only the store_body function of the provider "knows" what things
need to be done in this situation for this provider. So currently I see no
chance to handle this generically in mod_cache :/.


Regards

Rüdiger


Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Sat, Dec 10, 2005 at 04:51:22PM -0800, Roy Fielding wrote:
> Oy, crikey, I don't want that -- the proxy handler doesn't have the
> information needed to complete the response.  As I said before, the
> right way to do it is to send the error downstream.  There is no
> need to leave the filter in an error state.
> 
> As it turns out, we also have a request_rec->no_cache variable that
> is supposed to be set in this case.  I am curious whether the following
> patch will work (after reverting r355823).

That will get mod_cache not to pick up on it, right.  However, it won't
force the connection to close.  Or, give anything a chance to close it: be
it HTTP/1.1 or Waka.  =)  -- justin

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 05:59 PM, Justin Erenkrantz wrote:
> On Sun, Dec 11, 2005 at 03:28:00AM +0100, Ruediger Pluem wrote:
> 
>>>The connection is still kept open rather than being closed.  I can step
>>>through a trace with you if you like.  -- justin
>>
>>Hm. But why? Once we have returned from ap_process_request ap_process_http_connection
>>is left and the connection should be closed by the mpm.
> 
> 
> No clue.  That's why we need a debugger.  =)  Are you up for some virtual
> hackathon-ing in IRC?  -- justin


Attached an updated version of Roy's patch that now closes the connection (at least in my tests :-)).
The problem with Roys patch has been that the EOS bucket was never seen inside the loop where
he checked for the EOS bucket, because the loop only seems to run over the data / metadata buckets
just before the EOS bucket.

Regards

Rüdiger


Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Sun, Dec 11, 2005 at 03:28:00AM +0100, Ruediger Pluem wrote:
> > The connection is still kept open rather than being closed.  I can step
> > through a trace with you if you like.  -- justin
> 
> Hm. But why? Once we have returned from ap_process_request ap_process_http_connection
> is left and the connection should be closed by the mpm.

No clue.  That's why we need a debugger.  =)  Are you up for some virtual
hackathon-ing in IRC?  -- justin

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 03:12 AM, Justin Erenkrantz wrote:
> On Sat, Dec 10, 2005 at 05:49:10PM -0800, Roy Fielding wrote:
> 
>>+                    if (APR_BUCKET_IS_EOS(bucket) && bucket->data) {
>>+                        /* stream aborted and we have not ended it  
>>yet */
>>+                        c->keepalive = AP_CONN_CLOSE;
>>+                        bucket->data = NULL;
>>+                    }
> 
> 
> The connection is still kept open rather than being closed.  I can step
> through a trace with you if you like.  -- justin

Hm. But why? Once we have returned from ap_process_request ap_process_http_connection
is left and the connection should be closed by the mpm.

Regards

Rüdiger

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Sat, Dec 10, 2005 at 05:49:10PM -0800, Roy Fielding wrote:
> +                    if (APR_BUCKET_IS_EOS(bucket) && bucket->data) {
> +                        /* stream aborted and we have not ended it  
> yet */
> +                        c->keepalive = AP_CONN_CLOSE;
> +                        bucket->data = NULL;
> +                    }

The connection is still kept open rather than being closed.  I can step
through a trace with you if you like.  -- justin

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 02:49 AM, Roy T. Fielding wrote:
> On Dec 10, 2005, at 5:30 PM, Ruediger Pluem wrote:
> 

[..cut..]

> 
> 
> Ideally it would be in APR, but that runs into version problems.  Since
> it is a general streams issue, I would put the function in httpd.h, but
> right now I just want to know if it works at all (since I have no way
> of testing it).

Although I would like Brian to do a final test, it seems to work. I reverse
proxied to a Tomcat running the following jsp:

<%@ page import="java.util.Date"%><%
    Date datum;

    datum = new Date();
    response.setDateHeader("Expires",datum.getTime()+60000);
%>
<html>
<head>
<title>
Test
</title>
</head>
<body>
Start<br/>
<%

 long i;

 for (i=0; i < 10000; i++) {
%>
test<br/>
<%
out.flush();
Thread.sleep(100);
}
%>
Finish<br/>
</html>

via

ProxyPass /test balancer://mycluster/test stickysession=JSESSIONID nofailover=On

<Proxy balancer://mycluster>
BalancerMember http://127.0.0.1:8080 route=test
</Proxy>

And just killed the Tomcat via kill -9 before the answer was complete.
httpd correctly discards the content cached so far now.

Regards

Rüdiger

[..cut..]

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Brian Akins <br...@turner.com>.
Ruediger Pluem wrote:

> I hope Brian can step in and test it.
Currently, we are testing the c->aborted and return DONE patches.  It 
will be next week before I can test further patches as I have to roll 
out a fixed version of 2.2.0.


-- 
Brian Akins
Lead Systems Engineer
CNN Internet Technologies

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 03:03 AM, Ruediger Pluem wrote:

[..cut..]

> 
> BTW: Where is the extension of the request_rec structure?
> And isn't it a problem to change this structure in a stable branch, if
> we want to backport this fix?

Sorry my fault. I got the impression that no_cache needed to be added to the
request_rec, but it is already there :-).

Regards

Rüdiger

Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 02:49 AM, Roy T. Fielding wrote:
> On Dec 10, 2005, at 5:30 PM, Ruediger Pluem wrote:
> 
>> But with your approach you need to double the code in *each* scheme 
>> handler.
> 
> 
> mod_mem_cache doesn't look like it works right now.  I haven't looked
> at the other schemes yet.

I was talking about the proxy scheme handlers :-).
mod_proxy_ajp currently has the same problem that we are trying to fix in
mod_proxy_http

[..cut..]

> 
> 
> Ideally it would be in APR, but that runs into version problems.  Since
> it is a general streams issue, I would put the function in httpd.h, but

And where do you want to put the code? Or do you want to make it a macro?

> right now I just want to know if it works at all (since I have no way
> of testing it).

I hope Brian can step in and test it. Meanwhile I will have a look, if
I can do a test. For ease of testing: Could you please resent the mail
with the patch as attachment?

BTW: Where is the extension of the request_rec structure?
And isn't it a problem to change this structure in a stable branch, if
we want to backport this fix?

Regards

Rüdiger


Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Dec 10, 2005, at 5:30 PM, Ruediger Pluem wrote:
> But with your approach you need to double the code in *each* scheme  
> handler.

mod_mem_cache doesn't look like it works right now.  I haven't looked
at the other schemes yet.

> r355823 tries to avoid this by doing it a little later in the proxy  
> handler.
> I currently comment only on the position *where* you do it and not  
> what.
> But of course in the proxy handler I do not have rv add hand any  
> longer.
> The remaining part of the patch could be done also in the proxy  
> handler
> with e->data set to an int that is non null.
> Ok. Other approach would be to put your code in a function of  
> proxy_util.c
> where every scheme handler can call it. That sounds also like a  
> good solution.

Ideally it would be in APR, but that runs into version problems.  Since
it is a general streams issue, I would put the function in httpd.h, but
right now I just want to know if it works at all (since I have no way
of testing it).

I think this is the full patch, but there may be a better place to
check for EOS errors.

....Roy

Index: server/core_filters.c
===================================================================
--- server/core_filters.c	(revision 355728)
+++ server/core_filters.c	(working copy)
@@ -678,6 +678,11 @@
              for (i = offset; i < nvec; ) {
                  apr_bucket *bucket = APR_BRIGADE_FIRST(bb);
                  if (APR_BUCKET_IS_METADATA(bucket)) {
+                    if (APR_BUCKET_IS_EOS(bucket) && bucket->data) {
+                        /* stream aborted and we have not ended it  
yet */
+                        c->keepalive = AP_CONN_CLOSE;
+                        bucket->data = NULL;
+                    }
                      APR_BUCKET_REMOVE(bucket);
                      apr_bucket_destroy(bucket);
                  }
Index: modules/proxy/mod_proxy_http.c
===================================================================
--- modules/proxy/mod_proxy_http.c	(revision 355728)
+++ modules/proxy/mod_proxy_http.c	(working copy)
@@ -1481,12 +1481,16 @@
                      }
                      else if (rv != APR_SUCCESS) {
                          /* In this case, we are in real trouble  
because
-                         * our backend bailed on us, so abort our
-                         * connection to our user too.
+                         * our backend bailed on us.
                           */
                          ap_log_cerror(APLOG_MARK, APLOG_ERR, rv, c,
                                        "proxy: error reading  
response");
-                        c->aborted = 1;
+                        r->no_cache = 1;
+                        e = apr_bucket_eos_create(c->bucket_alloc);
+                        e->data = &rv;
+                        APR_BRIGADE_INSERT_TAIL(bb, e);
+                        ap_pass_brigade(r->output_filters, bb);
+                        backend->close = 1;
                          break;
                      }
                      /* next time try a non-blocking read */
Index: modules/cache/mod_disk_cache.c
===================================================================
--- modules/cache/mod_disk_cache.c	(revision 355728)
+++ modules/cache/mod_disk_cache.c	(working copy)
@@ -1010,7 +1010,7 @@
       * sanity checks.
       */
      if (APR_BUCKET_IS_EOS(APR_BRIGADE_LAST(bb))) {
-        if (r->connection->aborted) {
+        if (r->connection->aborted || r->no_cache) {
              ap_log_error(APLOG_MARK, APLOG_INFO, 0, r->server,
                           "disk_cache: Discarding body for URL %s "
                           "because connection has been aborted.",


Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/11/2005 01:51 AM, Roy T. Fielding wrote:
> On Dec 10, 2005, at 4:15 PM, rpluem@apache.org wrote:
> 
> 
> 
> Oy, crikey, I don't want that -- the proxy handler doesn't have the
> information needed to complete the response.  As I said before, the
> right way to do it is to send the error downstream.  There is no
> need to leave the filter in an error state.
> 
> As it turns out, we also have a request_rec->no_cache variable that
> is supposed to be set in this case.  I am curious whether the following
> patch will work (after reverting r355823).

But with your approach you need to double the code in *each* scheme handler.
r355823 tries to avoid this by doing it a little later in the proxy handler.
I currently comment only on the position *where* you do it and not what.
But of course in the proxy handler I do not have rv add hand any longer.
The remaining part of the patch could be done also in the proxy handler
with e->data set to an int that is non null.
Ok. Other approach would be to put your code in a function of proxy_util.c
where every scheme handler can call it. That sounds also like a good solution.

Regards

Rüdiger


Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Dec 10, 2005, at 4:15 PM, rpluem@apache.org wrote:

> Author: rpluem
> Date: Sat Dec 10 16:15:27 2005
> New Revision: 355823
>
> URL: http://svn.apache.org/viewcvs?rev=355823&view=rev
> Log:
> * Move handling of backends that broke after the headers have been  
> sent
>   into the proxy handler of mod_proxy.
>
>   This patch still sets r->connection->aborted to 1 which is currently
>   vetoed by Roy. Moving it from the scheme handler to the proxy  
> handler
>   should ease the reimplementation of this, as the scheme handlers  
> only
>   needs to return PROXY_BACKEND_BROKEN to signal the above  
> situation to
>   the proxy handler.

Oy, crikey, I don't want that -- the proxy handler doesn't have the
information needed to complete the response.  As I said before, the
right way to do it is to send the error downstream.  There is no
need to leave the filter in an error state.

As it turns out, we also have a request_rec->no_cache variable that
is supposed to be set in this case.  I am curious whether the following
patch will work (after reverting r355823).

....Roy


Index: modules/proxy/mod_proxy_http.c
===================================================================
--- modules/proxy/mod_proxy_http.c	(revision 355728)
+++ modules/proxy/mod_proxy_http.c	(working copy)
@@ -1481,12 +1481,16 @@
                      }
                      else if (rv != APR_SUCCESS) {
                          /* In this case, we are in real trouble  
because
-                         * our backend bailed on us, so abort our
-                         * connection to our user too.
+                         * our backend bailed on us.
                           */
                          ap_log_cerror(APLOG_MARK, APLOG_ERR, rv, c,
                                        "proxy: error reading  
response");
-                        c->aborted = 1;
+                        r->no_cache = 1;
+                        e = apr_bucket_eos_create(c->bucket_alloc);
+                        e->data = &rv;
+                        APR_BRIGADE_INSERT_TAIL(bb, e);
+                        ap_pass_brigade(r->output_filters, bb);
+                        backend->close = 1;
                          break;
                      }
                      /* next time try a non-blocking read */
Index: modules/cache/mod_disk_cache.c
===================================================================
--- modules/cache/mod_disk_cache.c	(revision 355728)
+++ modules/cache/mod_disk_cache.c	(working copy)
@@ -1010,7 +1010,7 @@
       * sanity checks.
       */
      if (APR_BUCKET_IS_EOS(APR_BRIGADE_LAST(bb))) {
-        if (r->connection->aborted) {
+        if (r->connection->aborted || r->no_cache) {
              ap_log_error(APLOG_MARK, APLOG_INFO, 0, r->server,
                           "disk_cache: Discarding body for URL %s "
                           "because connection has been aborted.",




Re: svn commit: r355823 - in /httpd/httpd/trunk/modules/proxy: mod_proxy.c mod_proxy.h mod_proxy_http.c

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Sun, Dec 11, 2005 at 12:15:33AM -0000, rpluem@apache.org wrote:
> Author: rpluem
> Date: Sat Dec 10 16:15:27 2005
> New Revision: 355823
> 
> URL: http://svn.apache.org/viewcvs?rev=355823&view=rev
> Log:
> * Move handling of backends that broke after the headers have been sent
>   into the proxy handler of mod_proxy.
> 
>   This patch still sets r->connection->aborted to 1 which is currently
>   vetoed by Roy. Moving it from the scheme handler to the proxy handler
>   should ease the reimplementation of this, as the scheme handlers only
>   needs to return PROXY_BACKEND_BROKEN to signal the above situation to
>   the proxy handler.

Note that the previous code would return DONE to the core.  This change
drops that.  As the code is written now, we're indicating to the core
that more data can be written; which would be badness.  -- justin