You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Mark Nottingham <mn...@mnot.net> on 2010/04/23 02:40:59 UTC
HTTP trailers?
I couldn't find any obvious way to set HTTP trailers in Apache 2.x without taking over all response processing (a la nph).
Did I miss something?
Cheers,
--
Mark Nottingham http://www.mnot.net/
Re: HTTP trailers?
Posted by Mark Nottingham <mn...@mnot.net>.
Yes; see
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
Cheers,
On 23/04/2010, at 11:44 AM, Sander Temme wrote:
> Mark,
>
> On Apr 22, 2010, at 5:40 PM, Mark Nottingham wrote:
>
>> I couldn't find any obvious way to set HTTP trailers in Apache 2.x without taking over all response processing (a la nph).
>
> Stupid question: what is an HTTP trailer? Is this in the context of Chunked transfer-encoding?
>
> S.
>
> --
> Sander Temme
> sctemme@apache.org
> PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
>
>
>
--
Mark Nottingham http://www.mnot.net/
Re: HTTP trailers?
Posted by Sander Temme <sc...@apache.org>.
Mark,
On Apr 22, 2010, at 5:40 PM, Mark Nottingham wrote:
> I couldn't find any obvious way to set HTTP trailers in Apache 2.x without taking over all response processing (a la nph).
Stupid question: what is an HTTP trailer? Is this in the context of Chunked transfer-encoding?
S.
--
Sander Temme
sctemme@apache.org
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
RE: HTTP trailers?
Posted by "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>.
> -----Original Message-----
> From: Jeff Trawick
I think there is a bigger design discussion that should
> happen, but I might have a patch down the road as a starter
> if all goes well at work.
>
> This is probably naive, but something to take pot shots at...
>
> use r->trailers_out
>
> non-empty r->trailers_out could force chunked encoding, as well as
> convey the trailers themselves
>
> 500 error if trailers set by the time chunking decision is made and
> caller can't handle
>
> consider logging msg if trailers found after response already started
> but trailers can't be sent for some reason (i.e., output filter adds
> trailer but trailers can't be sent)
>
> suppress-trailers or similar envvar probably needed for
> problematic clients
>
+1. Sounds reasonable.
Regards
Rüdiger
Re: HTTP trailers?
Posted by Jeff Trawick <tr...@gmail.com>.
On Fri, Apr 23, 2010 at 11:25 AM, Brian J. France <br...@brianfrance.com> wrote:
>
> On Apr 23, 2010, at 10:08 AM, William A. Rowe Jr. wrote:
>
>> On 4/23/2010 9:03 AM, Brian J. France wrote:
>>>
>>> You can build a module that is able to insert a trailer by adding a filter and ap_hook_create_request call.
>>
>> But doesn't this defeat the purpose of using a modular server
>> architecture? It seems this should be a facility of the core HTTP
>> filter, if anyone wants to offer the patch for 2.3.
>
>
> I agree, my module was more of a proof of concept that I can do it and then get some other server to able able to use it.
>
> Not sure what the best solution would be because multiple things need to happen. First part is you have to force chunk encoding either by removing content_length filter or tweaking the code to not add it if doing a trailer (which you might not know until it is time to insert a tailer).
>
> Then you have to tweak modules/http/chunk_filter.c to allow others to insert a trailer, like adding a ap_hook_http_trailer or a optional function for inserting it. I don't know if multiple modules should be allowed to add a trailer, if you do how to you join them since a trailer is nothing but a string ending with ASCII_CRLF (just strcat?). Should we just grab r->notes['http_trailer'] and let modules just add/set/append values?
>
> I think there is a bigger design discussion that should happen, but I might have a patch down the road as a starter if all goes well at work.
This is probably naive, but something to take pot shots at...
use r->trailers_out
non-empty r->trailers_out could force chunked encoding, as well as
convey the trailers themselves
500 error if trailers set by the time chunking decision is made and
caller can't handle
consider logging msg if trailers found after response already started
but trailers can't be sent for some reason (i.e., output filter adds
trailer but trailers can't be sent)
suppress-trailers or similar envvar probably needed for problematic clients
Re: HTTP trailers?
Posted by Mark Nottingham <mn...@mnot.net>.
Just to kick this discussion a bit;
The use cases that I've come across:
- adding debug / trace information to responses (e.g., how long it took, resource consumption, errors encountered)
- adding a late-bound ETag or Last-Modified to the response (so that you don't have to buffer)
- adding Content-MD5 or perhaps even something cryptographically signed (ditto)
I've had discussions with a few browser folks who have shown interest in making trailers available on their side (e.g., showing trace information in Firebug).
I don't agree that Apache should 5xx if trailers are set and they aren't able to be sent (e.g., because of a HTTP/1.0 client); the semantics of trailers in HTTP/1.1 are that they have to be able to be ignored by clients anyway (unless TE: trailers is received in the request, which IME is very uncommon). Dropping them on the floor is a fine solution -- as long as the code inserting the trailers knows this.
In the long run, it would also be interesting to have Apache examine the TE request header to determine whether trailers are supported; if they aren't, it could buffer the response and put the trailers up into the headers transparently. Of course, large responses might make this impractical, but in some cases it could be useful.
Finally -- HTTPbis gives us an opportunity to refine the design of trailers if there are issues.
Cheers,
On 24/04/2010, at 2:01 AM, William A. Rowe Jr. wrote:
> On 4/23/2010 10:25 AM, Brian J. France wrote:
>>
>> On Apr 23, 2010, at 10:08 AM, William A. Rowe Jr. wrote:
>>
>>> On 4/23/2010 9:03 AM, Brian J. France wrote:
>>>>
>>>> You can build a module that is able to insert a trailer by adding a filter and ap_hook_create_request call.
>>>
>>> But doesn't this defeat the purpose of using a modular server
>>> architecture? It seems this should be a facility of the core HTTP
>>> filter, if anyone wants to offer the patch for 2.3.
>>
>>
>> I agree, my module was more of a proof of concept that I can do it and then get some other server to able able to use it.
>
> :)
>
>> Not sure what the best solution would be because multiple things need to happen. First part is you have to force chunk encoding either by removing content_length filter or tweaking the code to not add it if doing a trailer (which you might not know until it is time to insert a tailer).
>
> Well, you also have to insert the 'Trailers' header, which must be known at the
> beginning of the request, so that becomes a simple trigger for dropping the
> content-length and forcing chunked encoding.
>
> "If no Trailer header field is present, the trailer SHOULD NOT include any header
> fields" is a very explicit statement :)
>
> This could be constructed from r->trailers_out, however users need to understand
> that after the beginning of the response, r->trailers out cannot be extended, only
> modified.
>
>> Then you have to tweak modules/http/chunk_filter.c to allow others to insert a trailer, like adding a ap_hook_http_trailer or a optional function for inserting it. I don't know if multiple modules should be allowed to add a trailer, if you do how to you join them since a trailer is nothing but a string ending with ASCII_CRLF (just strcat?). Should we just grab r->notes['http_trailer'] and let modules just add/set/append values?
>>
>> I think there is a bigger design discussion that should happen, but I might have a patch down the road as a starter if all goes well at work.
>
> These pieces seem more like implementation details.
--
Mark Nottingham http://www.mnot.net/
Re: HTTP trailers?
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 4/23/2010 10:25 AM, Brian J. France wrote:
>
> On Apr 23, 2010, at 10:08 AM, William A. Rowe Jr. wrote:
>
>> On 4/23/2010 9:03 AM, Brian J. France wrote:
>>>
>>> You can build a module that is able to insert a trailer by adding a filter and ap_hook_create_request call.
>>
>> But doesn't this defeat the purpose of using a modular server
>> architecture? It seems this should be a facility of the core HTTP
>> filter, if anyone wants to offer the patch for 2.3.
>
>
> I agree, my module was more of a proof of concept that I can do it and then get some other server to able able to use it.
:)
> Not sure what the best solution would be because multiple things need to happen. First part is you have to force chunk encoding either by removing content_length filter or tweaking the code to not add it if doing a trailer (which you might not know until it is time to insert a tailer).
Well, you also have to insert the 'Trailers' header, which must be known at the
beginning of the request, so that becomes a simple trigger for dropping the
content-length and forcing chunked encoding.
"If no Trailer header field is present, the trailer SHOULD NOT include any header
fields" is a very explicit statement :)
This could be constructed from r->trailers_out, however users need to understand
that after the beginning of the response, r->trailers out cannot be extended, only
modified.
> Then you have to tweak modules/http/chunk_filter.c to allow others to insert a trailer, like adding a ap_hook_http_trailer or a optional function for inserting it. I don't know if multiple modules should be allowed to add a trailer, if you do how to you join them since a trailer is nothing but a string ending with ASCII_CRLF (just strcat?). Should we just grab r->notes['http_trailer'] and let modules just add/set/append values?
>
> I think there is a bigger design discussion that should happen, but I might have a patch down the road as a starter if all goes well at work.
These pieces seem more like implementation details.
Re: HTTP trailers?
Posted by "Brian J. France" <br...@brianfrance.com>.
On Apr 23, 2010, at 10:08 AM, William A. Rowe Jr. wrote:
> On 4/23/2010 9:03 AM, Brian J. France wrote:
>>
>> You can build a module that is able to insert a trailer by adding a filter and ap_hook_create_request call.
>
> But doesn't this defeat the purpose of using a modular server
> architecture? It seems this should be a facility of the core HTTP
> filter, if anyone wants to offer the patch for 2.3.
I agree, my module was more of a proof of concept that I can do it and then get some other server to able able to use it.
Not sure what the best solution would be because multiple things need to happen. First part is you have to force chunk encoding either by removing content_length filter or tweaking the code to not add it if doing a trailer (which you might not know until it is time to insert a tailer).
Then you have to tweak modules/http/chunk_filter.c to allow others to insert a trailer, like adding a ap_hook_http_trailer or a optional function for inserting it. I don't know if multiple modules should be allowed to add a trailer, if you do how to you join them since a trailer is nothing but a string ending with ASCII_CRLF (just strcat?). Should we just grab r->notes['http_trailer'] and let modules just add/set/append values?
I think there is a bigger design discussion that should happen, but I might have a patch down the road as a starter if all goes well at work.
Brian
Re: HTTP trailers?
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 4/23/2010 9:03 AM, Brian J. France wrote:
>
> You can build a module that is able to insert a trailer by adding a filter and ap_hook_create_request call.
But doesn't this defeat the purpose of using a modular server
architecture? It seems this should be a facility of the core HTTP
filter, if anyone wants to offer the patch for 2.3.
Re: HTTP trailers?
Posted by "Brian J. France" <br...@brianfrance.com>.
For those following this thread, here is a update.
You can build a module that is able to insert a trailer by adding a filter and ap_hook_create_request call.
You have to have a ap_hook_create_request call that runs after http_core.c and loops over the r->output_filters and removes ap_content_length_filter_handle.
Then you have a AP_FTYPE_TRANSCODE+1 filter (chunk filter is AP_FTYPE_TRANSCODE and you want to be after) that loops over the brigade looking for a bucket that is:
ASCII_ZERO ASCII_CRLF ASCII_CRLF
followed by a APR_BUCKET_IS_EOS bucket.
Remove it and replaces it with:
ASCII_ZERO ASCII_CRLF <tailer string> ASCII_CRLF
and you have chunked encoding with a trailer.
Brian
On Apr 22, 2010, at 10:39 PM, Brian J. France wrote:
> On Apr 22, 2010, at 8:40 PM, Mark Nottingham wrote:
>> I couldn't find any obvious way to set HTTP trailers in Apache 2.x without taking over all response processing (a la nph).
>>
>> Did I miss something?
>
>
> I started hacking on this at work, but got the point where I can't insert a filter in the right spot.
>
> The problem I ran into is that if the Content-Length header is set (which ap_content_length_filter_handle sets), then the chunking filter doesn't kick in. Something about wanting to use sendfile so don't chunk.
>
> I can't get a filter inserted after ap_content_length_filter_handle, but before ap_http_header_filter (which adds the chunking filter).
>
> My plan was to insert a filter that removes the Content-Length header so the response will be chunked and add another filter after the chunking filter that would search for ASCII_ZERO ASCII_CRLF ASCII_CRLF and insert the trailer data before the last ASCII_CRLF. This was just a proof of concept to see if it would work and output data in the right format.
>
> If it all worked my plan was add a trailers hook and call the hook in chunk_filter.c in place of the /* <trailers> */ comment.
>
> Brian
>
> After typing this up and working through it again, I think I can add a filter AP_FTYPE_TRANSCODE+1 for the after chunking filter and AP_FTYPE_PROTOCOL+1 and get them to be in the right order. Will test that out now.
>
Re: HTTP trailers?
Posted by "Brian J. France" <br...@brianfrance.com>.
On Apr 22, 2010, at 8:40 PM, Mark Nottingham wrote:
> I couldn't find any obvious way to set HTTP trailers in Apache 2.x without taking over all response processing (a la nph).
>
> Did I miss something?
I started hacking on this at work, but got the point where I can't insert a filter in the right spot.
The problem I ran into is that if the Content-Length header is set (which ap_content_length_filter_handle sets), then the chunking filter doesn't kick in. Something about wanting to use sendfile so don't chunk.
I can't get a filter inserted after ap_content_length_filter_handle, but before ap_http_header_filter (which adds the chunking filter).
My plan was to insert a filter that removes the Content-Length header so the response will be chunked and add another filter after the chunking filter that would search for ASCII_ZERO ASCII_CRLF ASCII_CRLF and insert the trailer data before the last ASCII_CRLF. This was just a proof of concept to see if it would work and output data in the right format.
If it all worked my plan was add a trailers hook and call the hook in chunk_filter.c in place of the /* <trailers> */ comment.
Brian
After typing this up and working through it again, I think I can add a filter AP_FTYPE_TRANSCODE+1 for the after chunking filter and AP_FTYPE_PROTOCOL+1 and get them to be in the right order. Will test that out now.