You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Yann Ylavic <yl...@gmail.com> on 2013/05/22 13:06:01 UTC

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

On 05/14/2013 08:58 PM, minfrin@apache.org wrote:
> Author: minfrin
> Date: Tue May 21 16:10:02 2013
> New Revision: 1484852
>
> URL: http://svn.apache.org/r1484852
> Log:
> core: Remove apr_brigade_flatten(), buffering and duplicated code
> from the HTTP_IN filter, parse chunks in a single pass with zero copy.
> Reduce memory usage by 48 bytes per request.
>
> Modified:
> httpd/httpd/trunk/CHANGES
> httpd/httpd/trunk/modules/http/http_filters.c

Shouldn't the BODY_CHUNK_END state only accept [CR]LF ?
Currently anything but a LF is ignored.

Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c    (revision 1485126)
+++ modules/http/http_filters.c    (working copy)
@@ -100,9 +100,15 @@

         /* handle CRLF after the chunk */
         if (ctx->state == BODY_CHUNK_END) {
-            if (c == LF) {
-                ctx->state = BODY_CHUNK;
+            if (c == CR && ++i < len) {
+                c = buffer[i];
+                ap_xlate_proto_from_ascii(&c, 1);
             }
+            if (c != LF) {
+                /* [CR]LF expected! */
+                return APR_EGENERAL;
+            }
+            ctx->state = BODY_CHUNK;
             i++;
             continue;
         }

Regards,
Yann.

RE: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>.

From: Yann Ylavic [mailto:ylavic.dev@gmail.com]
Sent: Mittwoch, 22. Mai 2013 13:06
To: dev@httpd.apache.org
Subject: Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

On 05/14/2013 08:58 PM, minfrin@apache.org<ma...@apache.org> wrote:
> Author: minfrin
> Date: Tue May 21 16:10:02 2013
> New Revision: 1484852
>
> URL: http://svn.apache.org/r1484852
> Log:
> core: Remove apr_brigade_flatten(), buffering and duplicated code
> from the HTTP_IN filter, parse chunks in a single pass with zero copy.
> Reduce memory usage by 48 bytes per request.
>
> Modified:
> httpd/httpd/trunk/CHANGES
> httpd/httpd/trunk/modules/http/http_filters.c
Shouldn't the BODY_CHUNK_END state only accept [CR]LF ?
Currently anything but a LF is ignored.

Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c    (revision 1485126)
+++ modules/http/http_filters.c    (working copy)
@@ -100,9 +100,15 @@

         /* handle CRLF after the chunk */
         if (ctx->state == BODY_CHUNK_END) {
-            if (c == LF) {
-                ctx->state = BODY_CHUNK;
+            if (c == CR && ++i < len) {
+                c = buffer[i];
+                ap_xlate_proto_from_ascii(&c, 1);
             }
+            if (c != LF) {
+                /* [CR]LF expected! */
+                return APR_EGENERAL;
+            }
+            ctx->state = BODY_CHUNK;
             i++;
             continue;
         }
Regards,
Yann.

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Graham Leggett <mi...@sharp.fm>.
On 23 May 2013, at 9:58 AM, "Plüm, Rüdiger, Vodafone Group" <ru...@vodafone.com> wrote:

> Shouldn't we apply the following patch as well to be save that we have a sane state
> of b at the beginning of the loop and no remaining buckets e.g. metabuckets left in it?

We should, fixed in r1485772.

Regards,
Graham
--


RE: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>.
Shouldn't we apply the following patch as well to be save that we have a sane state
of b at the beginning of the loop and no remaining buckets e.g. metabuckets left in it?


Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c (revision 1485262)
+++ modules/http/http_filters.c (working copy)
@@ -331,9 +331,9 @@
         APR_BRIGADE_INSERT_TAIL(b, e);
         return APR_SUCCESS;
     }
-    apr_brigade_cleanup(b);

     do {
+        apr_brigade_cleanup(b);
         again = 0; /* until further notice */

         /* read and handle the brigade */


Regards

Rüdiger

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Graham Leggett <mi...@sharp.fm>.
On 22 May 2013, at 4:19 PM, Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com> wrote:

> Yeah. You are correct. Looks sufficient.

Fixed in r1485257.

Regards,
Graham
--


RE: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>.

> -----Original Message-----
> From: Graham Leggett [mailto:minfrin@sharp.fm]
> Sent: Mittwoch, 22. Mai 2013 16:17
> To: dev@httpd.apache.org
> Subject: Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES
> modules/http/http_filters.c
> 
> On 22 May 2013, at 3:57 PM, Plüm, Rüdiger, Vodafone Group
> <ru...@vodafone.com> wrote:
> 
> > I guess you missed a special case:
> >
> > If the reverse proxy backend sends a response without a Content-Length
> header and without a Transfer-Encoding,
> > which is IMHO valid for HTTP 1.0 responses if the connection is closed
> after the response.
> > The following patch would fix this:
> >
> > Index: modules/http/http_filters.c
> > ===================================================================
> > --- modules/http/http_filters.c (revision 1485192)
> > +++ modules/http/http_filters.c (working copy)
> > @@ -394,8 +394,15 @@
> >        case BODY_LENGTH:
> >        case BODY_CHUNK_DATA: {
> >
> > -            /* Ensure that the caller can not go over our boundary
> point. */
> > -            if (ctx->remaining < readbytes) {
> > +            /*
> > +             * Ensure that the caller can not go over our boundary
> point,
> > +             * BUT only if this is not a response by reverse proxy
> backend
> > +             * that sent no Content-Length header and has no transfer
> encoding
> > +             * which is valid for a non keep-alive HTTP 1.0 response.
> > +             */
> > +            if ((ctx->remaining < readbytes) && !((ctx->remaining ==
> 0) &&
> > +                  (ctx->state == BODY_NONE) &&
> > +                  (f->r->proxyreq == PROXYREQ_RESPONSE))) {
> >                readbytes = ctx->remaining;
> >            }
> >            if (readbytes > 0) {
> >
> > Should I commit it?
> 
> Hmmm. the check for f->r->proxyreq == PROXYREQ_RESPONSE shouldn't be
> necessary at this point, as we already check for this case here:
> 
>        /* If we don't have a request entity indicated by the headers,
> EOS.
>         * (BODY_NONE is a valid intermediate state due to trailers,
>         *  but it isn't a valid starting state.)
>         *
>         * RFC 2616 Section 4.4 note 5 states that connection-close
>         * is invalid for a request entity - request bodies must be
>         * denoted by C-L or T-E: chunked.
>         *
>         * Note that since the proxy uses this filter to handle the
>         * proxied *response*, proxy responses MUST be exempt.
>         */
>        if (ctx->state == BODY_NONE && f->r->proxyreq !=
> PROXYREQ_RESPONSE) {
>            e = apr_bucket_eos_create(f->c->bucket_alloc);
>            APR_BRIGADE_INSERT_TAIL(b, e);
>            ctx->eos_sent = 1;
>            return APR_SUCCESS;
>        }
> 
> The BODY_NONE case however shouldn't be constrained as you've said,
> something like this?
> 
> Index: modules/http/http_filters.c
> ===================================================================
> --- modules/http/http_filters.c	(revision 1484926)
> +++ modules/http/http_filters.c	(working copy)
> @@ -395,7 +395,7 @@
>         case BODY_CHUNK_DATA: {
> 
>             /* Ensure that the caller can not go over our boundary
> point. */
> -            if (ctx->remaining < readbytes) {
> +            if (ctx->state != BODY_NONE && ctx->remaining < readbytes)
> {
>                 readbytes = ctx->remaining;
>             }
>             if (readbytes > 0) {
> 

Yeah. You are correct. Looks sufficient.

Regards

Rüdiger


Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Graham Leggett <mi...@sharp.fm>.
On 22 May 2013, at 3:57 PM, Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com> wrote:

> I guess you missed a special case:
> 
> If the reverse proxy backend sends a response without a Content-Length header and without a Transfer-Encoding,
> which is IMHO valid for HTTP 1.0 responses if the connection is closed after the response.
> The following patch would fix this:
> 
> Index: modules/http/http_filters.c
> ===================================================================
> --- modules/http/http_filters.c (revision 1485192)
> +++ modules/http/http_filters.c (working copy)
> @@ -394,8 +394,15 @@
>        case BODY_LENGTH:
>        case BODY_CHUNK_DATA: {
> 
> -            /* Ensure that the caller can not go over our boundary point. */
> -            if (ctx->remaining < readbytes) {
> +            /*
> +             * Ensure that the caller can not go over our boundary point,
> +             * BUT only if this is not a response by reverse proxy backend
> +             * that sent no Content-Length header and has no transfer encoding
> +             * which is valid for a non keep-alive HTTP 1.0 response.
> +             */
> +            if ((ctx->remaining < readbytes) && !((ctx->remaining == 0) &&
> +                  (ctx->state == BODY_NONE) &&
> +                  (f->r->proxyreq == PROXYREQ_RESPONSE))) {
>                readbytes = ctx->remaining;
>            }
>            if (readbytes > 0) {
> 
> Should I commit it?

Hmmm… the check for f->r->proxyreq == PROXYREQ_RESPONSE shouldn't be necessary at this point, as we already check for this case here:

       /* If we don't have a request entity indicated by the headers, EOS.
        * (BODY_NONE is a valid intermediate state due to trailers,
        *  but it isn't a valid starting state.)
        *
        * RFC 2616 Section 4.4 note 5 states that connection-close
        * is invalid for a request entity - request bodies must be
        * denoted by C-L or T-E: chunked.
        *
        * Note that since the proxy uses this filter to handle the
        * proxied *response*, proxy responses MUST be exempt.
        */
       if (ctx->state == BODY_NONE && f->r->proxyreq != PROXYREQ_RESPONSE) {
           e = apr_bucket_eos_create(f->c->bucket_alloc);
           APR_BRIGADE_INSERT_TAIL(b, e);
           ctx->eos_sent = 1;
           return APR_SUCCESS;
       }

The BODY_NONE case however shouldn't be constrained as you've said, something like this?

Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c	(revision 1484926)
+++ modules/http/http_filters.c	(working copy)
@@ -395,7 +395,7 @@
        case BODY_CHUNK_DATA: {

            /* Ensure that the caller can not go over our boundary point. */
-            if (ctx->remaining < readbytes) {
+            if (ctx->state != BODY_NONE && ctx->remaining < readbytes) {
                readbytes = ctx->remaining;
            }
            if (readbytes > 0) {

Regards,
Graham
--


RE: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>.
I guess you missed a special case:

If the reverse proxy backend sends a response without a Content-Length header and without a Transfer-Encoding,
which is IMHO valid for HTTP 1.0 responses if the connection is closed after the response.
The following patch would fix this:

Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c (revision 1485192)
+++ modules/http/http_filters.c (working copy)
@@ -394,8 +394,15 @@
         case BODY_LENGTH:
         case BODY_CHUNK_DATA: {

-            /* Ensure that the caller can not go over our boundary point. */
-            if (ctx->remaining < readbytes) {
+            /*
+             * Ensure that the caller can not go over our boundary point,
+             * BUT only if this is not a response by reverse proxy backend
+             * that sent no Content-Length header and has no transfer encoding
+             * which is valid for a non keep-alive HTTP 1.0 response.
+             */
+            if ((ctx->remaining < readbytes) && !((ctx->remaining == 0) &&
+                  (ctx->state == BODY_NONE) &&
+                  (f->r->proxyreq == PROXYREQ_RESPONSE))) {
                 readbytes = ctx->remaining;
             }
             if (readbytes > 0) {

Should I commit it?

Regards

Rüdiger

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Yann Ylavic <yl...@gmail.com>.
Sorry, "linelimit" is an int and compared to an int.
Only "i" should be a size_t...

Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c    (revision 1485126)
+++ modules/http/http_filters.c    (working copy)
@@ -91,7 +91,7 @@
 static apr_status_t parse_chunk_size(http_ctx_t *ctx, const char *buffer,
         apr_size_t len, int linelimit)
 {
-    int i = 0;
+    apr_size_t i = 0;

     while (i < len) {
         char c = buffer[i];

Regards,
Yann.

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Yann Ylavic <yl...@gmail.com>.
parse_chunk_size() also compares int's and size_t's.

Index: modules/http/http_filters.c
===================================================================
--- modules/http/http_filters.c    (revision 1485126)
+++ modules/http/http_filters.c    (working copy)
@@ -89,9 +89,9 @@
  * In general, any negative number can be considered an overflow error.
  */
 static apr_status_t parse_chunk_size(http_ctx_t *ctx, const char *buffer,
-        apr_size_t len, int linelimit)
+        apr_size_t len, apr_size_t linelimit)
 {
-    int i = 0;
+    apr_size_t i = 0;

     while (i < len) {
         char c = buffer[i];
@@ -100,9 +100,15 @@

         /* handle CRLF after the chunk */
         if (ctx->state == BODY_CHUNK_END) {
-            if (c == LF) {
-                ctx->state = BODY_CHUNK;
+            if (c == CR && ++i < len) {
+                c = buffer[i];
+                ap_xlate_proto_from_ascii(&c, 1);
             }
+            if (c != LF) {
+                /* [CR]LF expected! */
+                return APR_EGENERAL;
+            }
+            ctx->state = BODY_CHUNK;
             i++;
             continue;
         }

Regards,
Yann.


On Wed, May 22, 2013 at 1:06 PM, Yann Ylavic <yl...@gmail.com> wrote:

> On 05/14/2013 08:58 PM, minfrin@apache.org wrote:
> > Author: minfrin
> > Date: Tue May 21 16:10:02 2013
> > New Revision: 1484852
> >
> > URL: http://svn.apache.org/r1484852
> > Log:
> > core: Remove apr_brigade_flatten(), buffering and duplicated code
> > from the HTTP_IN filter, parse chunks in a single pass with zero copy.
> > Reduce memory usage by 48 bytes per request.
> >
> > Modified:
> > httpd/httpd/trunk/CHANGES
> > httpd/httpd/trunk/modules/http/http_filters.c
>
> Shouldn't the BODY_CHUNK_END state only accept [CR]LF ?
> Currently anything but a LF is ignored.
>
> Index: modules/http/http_filters.c
> ===================================================================
> --- modules/http/http_filters.c    (revision 1485126)
> +++ modules/http/http_filters.c    (working copy)
> @@ -100,9 +100,15 @@
>
>          /* handle CRLF after the chunk */
>          if (ctx->state == BODY_CHUNK_END) {
> -            if (c == LF) {
> -                ctx->state = BODY_CHUNK;
> +            if (c == CR && ++i < len) {
> +                c = buffer[i];
> +                ap_xlate_proto_from_ascii(&c, 1);
>              }
> +            if (c != LF) {
> +                /* [CR]LF expected! */
> +                return APR_EGENERAL;
> +            }
> +            ctx->state = BODY_CHUNK;
>              i++;
>              continue;
>          }
>
> Regards,
> Yann.
>

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, May 22, 2013 at 3:04 PM, Graham Leggett <mi...@sharp.fm> wrote:

> All line lengths including the one you're referring to are constrained by
> LimitRequestFieldSize, so to say that the protocol is unconstrained is
> false.
>

Indeed I am wrong about the absence of constraint, which is not
LimitRequestFieldSize in this case (reading the CRLF after the chunk, not a
header), but the one of ap_get_brigade(..., AP_MODE_GETLINE, ..., 0)  =>
0???

I also agree this is not related to your commit since you didn't change the
existing behaviour, but I never looked at the chunking parser before, it
just jumped out at my eyes when trying to resolve my conflicting code :p

Thank you for your replies,
regards,
Yann.

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Graham Leggett <mi...@sharp.fm>.
On 22 May 2013, at 2:47 PM, Yann Ylavic <yl...@gmail.com> wrote:

> Well, one could inject arbitrary data in this room (with no LF), bypassing LimitRequestBody (which does not count chunks separators), and eat resources.
> This opens doors, as often when a protocol is not checked carefully…

Again, this matches the previous filter behaviour, and changing this is a separate discussion.

All line lengths including the one you're referring to are constrained by LimitRequestFieldSize, so to say that the protocol is unconstrained is false.

Regards,
Graham
--


Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, May 22, 2013 at 2:07 PM, Graham Leggett <mi...@sharp.fm> wrote:

> On 22 May 2013, at 1:48 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> > I am not saying that the filter should not tolerate LF only as sperator,
> but that after the chunk (ie. in the BODY_CHUNK_END state) it should only
> accept CRLF (or LF only) and nothing else.
>
> Sure, but the old filter behaved this way, and so the new filter has had
> to match it exactly. We just don't know what code is out there that depends
> on this behaviour.
>
>
Well, one could inject arbitrary data in this room (with no LF), bypassing
LimitRequestBody (which does not count chunks separators), and eat
resources.
This opens doors, as often when a protocol is not checked carefully...

Regards,
>

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Graham Leggett <mi...@sharp.fm>.
On 22 May 2013, at 1:48 PM, Yann Ylavic <yl...@gmail.com> wrote:

> I am not saying that the filter should not tolerate LF only as sperator, but that after the chunk (ie. in the BODY_CHUNK_END state) it should only accept CRLF (or LF only) and nothing else.

Sure, but the old filter behaved this way, and so the new filter has had to match it exactly. We just don't know what code is out there that depends on this behaviour.

Regards,
Graham
--


Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Yann Ylavic <yl...@gmail.com>.
I am not saying that the filter should not tolerate LF only as sperator,
but that after the chunk (ie. in the BODY_CHUNK_END state) it should only
accept CRLF (or LF only) and nothing else.

Regards,
Yann.


On Wed, May 22, 2013 at 1:31 PM, Graham Leggett <mi...@sharp.fm> wrote:

> On 22 May 2013, at 1:06 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> On 05/14/2013 08:58 PM, minfrin@apache.org wrote:
> > Author: minfrin
> > Date: Tue May 21 16:10:02 2013
> > New Revision: 1484852
> >
> > URL: http://svn.apache.org/r1484852
> > Log:
> > core: Remove apr_brigade_flatten(), buffering and duplicated code
> > from the HTTP_IN filter, parse chunks in a single pass with zero copy.
> > Reduce memory usage by 48 bytes per request.
> >
> > Modified:
> > httpd/httpd/trunk/CHANGES
> > httpd/httpd/trunk/modules/http/http_filters.c
>
> Shouldn't the BODY_CHUNK_END state only accept [CR]LF ?
> Currently anything but a LF is ignored.
>
>
> Yep, as is pointed out in this comment:
>
>         /* handle a chunk part, or a chunk extension */
>         /*
>          * In theory, we are supposed to expect CRLF only, but our
>          * test suite sends LF only. Tolerate a missing CR.
>          */
>         if (c == ';' || c == CR) {
>             ctx->state = BODY_CHUNK_EXT;
>         }
>
> Our test suite uses LF instead of CRLF as a chunk terminator in many
> cases, and we don't know what code in the wild depends on this behaviour.
> This isn't something we're going to fix easily.
>
> Regards,
> Graham
> --
>
>

Re: svn commit: r1484852 - in /httpd/httpd/trunk: CHANGES modules/http/http_filters.c

Posted by Graham Leggett <mi...@sharp.fm>.
On 22 May 2013, at 1:06 PM, Yann Ylavic <yl...@gmail.com> wrote:

> On 05/14/2013 08:58 PM, minfrin@apache.org wrote:
> > Author: minfrin 
> > Date: Tue May 21 16:10:02 2013 
> > New Revision: 1484852 
> >
> > URL: http://svn.apache.org/r1484852 
> > Log: 
> > core: Remove apr_brigade_flatten(), buffering and duplicated code 
> > from the HTTP_IN filter, parse chunks in a single pass with zero copy. 
> > Reduce memory usage by 48 bytes per request. 
> >
> > Modified: 
> > httpd/httpd/trunk/CHANGES 
> > httpd/httpd/trunk/modules/http/http_filters.c 
> 
> Shouldn't the BODY_CHUNK_END state only accept [CR]LF ?
> Currently anything but a LF is ignored.

Yep, as is pointed out in this comment:

        /* handle a chunk part, or a chunk extension */
        /*
         * In theory, we are supposed to expect CRLF only, but our
         * test suite sends LF only. Tolerate a missing CR.
         */
        if (c == ';' || c == CR) {
            ctx->state = BODY_CHUNK_EXT;
        }

Our test suite uses LF instead of CRLF as a chunk terminator in many cases, and we don't know what code in the wild depends on this behaviour. This isn't something we're going to fix easily.

Regards,
Graham
--