You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Geoff Millikan <gm...@t1shopper.com> on 2011/06/02 03:54:10 UTC
RE: [users@httpd] mod_deflate and chunked encoding
> In my case, it appears Apache selects the chunked encoding automagically
> when the size of the page goes over a certain size. Apache doesn't seem
> to set the smaller pages to the chunked encoding (even though they are gzipped).
No, this was wrong. Mod_file_cache was tricking me (again). Apache appears to set the Transfer-Encoding to chunked when the page is
actively being gzipped and the final gzip file is bigger than DeflateBufferSize. But if the page was gzipped and stored on disk by
mod_file_cache (like we do) it means Apache is merely spooling the file off the disk (it doesn't put it though gzip a second time)
and chunked encoding will *not* be used.
Now I'm going to speculate that a chunked gzip web page is going to be slower than the unchunked equivalent because the browser have
to spend "some amount" of time putting the chunked version back together. But I'm wrong again. We just tested this and the timing
of Firefox 4's load event didn't increase with the number of chunks. Even setting DeflateBufferSize down to 500 bytes on a 40K page
didn't produce consistent differences.
The one consistent metric was that with chunking on (DeflateBufferSize smaller than total compressed page size), Apache was able to
get the first byte to the web browser about 75ms faster but again, the faster response time didn't translate into a consistently
faster rendering of the page by the browser.
This testing needs further study before it's taken as gospel. We only ran about 4 tests per DeflateBufferSize.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
Oh by the way. Sorry for not being specific enough in my question. I
am not really familiar with Apache modules (except for some mod_perl
experience) and do not know how to word my question correctly.
I guess my original question was whether mod_deflate dechunks and
compresses on the fly. Response seems to be that mod_deflate is not
responsible for doing that anyway (correct?). So next question I
suppose is who's responsible for making an on the fly chunked
compression out of a chunked response. (Guess still not correctly
worded and that it depends on the module in the middle, but hope it is
closer. :)
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Stormy <st...@stormy.ca>.
At 10:56 PM 6/5/2011 +0200, Xavier Noria wrote:
>[snip] I mean. If it is true that Passenger should dechunk (as William says),
>but it is not doing that, but the client still gets the chunked
>response, I wondered whether it worked by luck rather than by all
>the pieces following the contracts. Always according to William's.
mod_deflate is a compression (input decomp possible) capability with client
caveats. Eric C. has given you the definitive response including streaming,
but maybe you should have a look at
<http://httpd.apache.org/docs/2.0/mod/mod_deflate.html>
I'm not an expert in MS <=> Java matters - but there's a group at
poi.apache.org that might help with "chunking" (but then again, I'm getting
long in the tooth and maybe have missed some of the latest innovations.)
Paul
Tired old sys-admin
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
On Sun, Jun 5, 2011 at 10:47 PM, Stormy <st...@stormy.ca> wrote:
> Ah, interesting... that you say it's an Apache module. Maybe Messrs Hongli
> Lai & Ninh Bui could help you with your compression and chunking challenges?
> 'Cos when you suggest that Apache is functioning "Guess that works by luck"
> I might be tempted to suggest that Apache is a little more mature than
> Phusion.
Oh no, I don't want to imply that.
I mean. If it is true that Passenger should dechunk (as William says),
but it is not doing that, but the client still gets the chunked
response, I wondered whether it worked by luck rather than by all
the pieces following the contracts. Always according to William's.
> But then again, your P.S. implies that you're a Phusion developer... ah,
> interesting.
I'm not, I am obviously not familiar with Apache development.
Though I am trying to get the picture about how this should work.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Stormy <st...@stormy.ca>.
At 09:15 PM 6/5/2011 +0200, Xavier Noria wrote:
>On Sun, Jun 5, 2011 at 8:16 PM, William A. Rowe Jr. <wr...@rowe-clan.net>
>wrote:
> > httpd (conditionally) handles the chunking... the app generator's
> > chunking is never used. Â What *module* is installed in httpd? Â I'm
> > not familiar with the above.
> >
> > Only the entry point to httpd, e.g. mod_proxy, mod_wl, mod_ws, mod_jk
> > mod_cgi or whatnot should ever dechunk. Â The response body must pass
> > through httpd without any protocol (chunking) or headers.
>
>Ah, interesting.
>
>Phusion Passenger is an Apache module itself:
>
> http://www.modrails.com/
Yes - ah, interesting... from the web site you quote:
"__Supports__ ... the industry standard Apache web server" [my emphasis]
"Zero maintenance. No ... server process monitoring ... "
"'Phusion' and 'Phusion Passenger' are trademarks of Hongli Lai & Ninh Bui.
All rights reserved."
Ah, interesting... that you say it's an Apache module. Maybe Messrs Hongli
Lai & Ninh Bui could help you with your compression and chunking
challenges? 'Cos when you suggest that Apache is functioning "Guess that
works by luck" I might be tempted to suggest that Apache is a little more
mature than Phusion.
But then again, your P.S. implies that you're a Phusion developer... ah,
interesting.
Best - Paul
Tired old sys-admin.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
On Sun, Jun 5, 2011 at 11:51 PM, Eric Covener <co...@gmail.com> wrote:
> On Sun, Jun 5, 2011 at 5:40 PM, Xavier Noria <fx...@hashref.com> wrote:
>> On Sun, Jun 5, 2011 at 11:36 PM, Eric Covener <co...@gmail.com> wrote:
>>
>>>> If Passenger has to dechunk, and we want a chunked compressed
>>>> response, and Apache is the one responsible for doing that, how should
>>>> we signal Apache that we want compression and streaming for that
>>>> particular response.
>>>
>>> This is the main path of just configuring mod_deflate. Nothing else
>>> required. Only passenger cares how passenger decided what the content
>>> of the response was, to everyone else it's just buckets of data.
>>>
>>> "Streaming" just means the module that generates the content of the
>>> response routinely hands some data off to Apache to send off to the
>>> client rather then e.g. waiting until it's 100% done. It's also not
>>> something you need to configure.
>>
>> But there has to be something else, because in streaming I want a
>> chunked response that should start sending chunks to the client as
>> soon as possible.
>>
>> If the back server sent just 200 bytes in the first chunk, I want
>> those 200 bytes compressed and forwarded to the client exactly now. In
>> particular there's no Content-Length computed.
>
> Does passenger pass the first 200 bytes to Apache?
>
> Does it follow them with a "flush bucket"?
>
> Does the protocol passenger uses on the backend allow your app to tell
> it when to flush?
>
> Does the result change when you turn on mod_deflate (the preceding
> questions are probably best asked on a passenger list, with this final
> questions answer in-hand)
Excellent! I'll ask that to the Passenger guys.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Eric Covener <co...@gmail.com>.
On Sun, Jun 5, 2011 at 5:40 PM, Xavier Noria <fx...@hashref.com> wrote:
> On Sun, Jun 5, 2011 at 11:36 PM, Eric Covener <co...@gmail.com> wrote:
>
>>> If Passenger has to dechunk, and we want a chunked compressed
>>> response, and Apache is the one responsible for doing that, how should
>>> we signal Apache that we want compression and streaming for that
>>> particular response.
>>
>> This is the main path of just configuring mod_deflate. Nothing else
>> required. Only passenger cares how passenger decided what the content
>> of the response was, to everyone else it's just buckets of data.
>>
>> "Streaming" just means the module that generates the content of the
>> response routinely hands some data off to Apache to send off to the
>> client rather then e.g. waiting until it's 100% done. It's also not
>> something you need to configure.
>
> But there has to be something else, because in streaming I want a
> chunked response that should start sending chunks to the client as
> soon as possible.
>
> If the back server sent just 200 bytes in the first chunk, I want
> those 200 bytes compressed and forwarded to the client exactly now. In
> particular there's no Content-Length computed.
Does passenger pass the first 200 bytes to Apache?
Does it follow them with a "flush bucket"?
Does the protocol passenger uses on the backend allow your app to tell
it when to flush?
Does the result change when you turn on mod_deflate (the preceding
questions are probably best asked on a passenger list, with this final
questions answer in-hand)
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
On Sun, Jun 5, 2011 at 11:36 PM, Eric Covener <co...@gmail.com> wrote:
>> If Passenger has to dechunk, and we want a chunked compressed
>> response, and Apache is the one responsible for doing that, how should
>> we signal Apache that we want compression and streaming for that
>> particular response.
>
> This is the main path of just configuring mod_deflate. Nothing else
> required. Only passenger cares how passenger decided what the content
> of the response was, to everyone else it's just buckets of data.
>
> "Streaming" just means the module that generates the content of the
> response routinely hands some data off to Apache to send off to the
> client rather then e.g. waiting until it's 100% done. It's also not
> something you need to configure.
But there has to be something else, because in streaming I want a
chunked response that should start sending chunks to the client as
soon as possible.
If the back server sent just 200 bytes in the first chunk, I want
those 200 bytes compressed and forwarded to the client exactly now. In
particular there's no Content-Length computed.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Eric Covener <co...@gmail.com>.
> If Passenger has to dechunk, and we want a chunked compressed
> response, and Apache is the one responsible for doing that, how should
> we signal Apache that we want compression and streaming for that
> particular response.
This is the main path of just configuring mod_deflate. Nothing else
required. Only passenger cares how passenger decided what the content
of the response was, to everyone else it's just buckets of data.
"Streaming" just means the module that generates the content of the
response routinely hands some data off to Apache to send off to the
client rather then e.g. waiting until it's 100% done. It's also not
something you need to configure.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
On Sun, Jun 5, 2011 at 9:54 PM, William A. Rowe Jr. <wr...@rowe-clan.net> wrote:
>> So I understand from your reply that httpd is the only one resposible
>> for chunked responses, compressed or otherwise. Is that correct?
>
> Thanks for the info. You are sort-of correct. The backend can possibly
> optimize things by using chunked httpd as its own transport.
>
>> So a module like Passenger is expected to dechunk and pass the body
>> to httpd? As of this writing chunked responses are passing through
>> Apache if compression is not involved. Guess that works by luck, but
>> it is still not playing by the rules?
>
> Exactly. I's put is as Passanger is expected to dechunk and pass the
> body *through* httpd.
>
> Try using mod_substitute/mod_sed on that response :)
I do not understand that one :).
If Passenger has to dechunk, and we want a chunked compressed
response, and Apache is the one responsible for doing that, how should
we signal Apache that we want compression and streaming for that
particular response.
Those modules you mentioned seem to do ordinary search and resplace.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 6/5/2011 2:15 PM, Xavier Noria wrote:
>
> Ah, interesting.
>
> Phusion Passenger is an Apache module itself:
>
> http://www.modrails.com/
>
> Passenger is the most used solution for production deployments in Ruby
> on Rails nowadays.
>
> So I understand from your reply that httpd is the only one resposible
> for chunked responses, compressed or otherwise. Is that correct?
Thanks for the info. You are sort-of correct. The backend can possibly
optimize things by using chunked httpd as its own transport.
> So a module like Passenger is expected to dechunk and pass the body
> to httpd? As of this writing chunked responses are passing through
> Apache if compression is not involved. Guess that works by luck, but
> it is still not playing by the rules?
Exactly. I's put is as Passanger is expected to dechunk and pass the
body *through* httpd.
Try using mod_substitute/mod_sed on that response :)
> If whether the response has to be streamed or not is decided by the
> application, which is the proper way to force httpd to stream, and
> eventually also compress?
Exactly.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
On Sun, Jun 5, 2011 at 8:16 PM, William A. Rowe Jr. <wr...@rowe-clan.net> wrote:
> On 6/5/2011 12:31 PM, Xavier Noria wrote:
>
> httpd (conditionally) handles the chunking... the app generator's
> chunking is never used. What *module* is installed in httpd? I'm
> not familiar with the above.
>
> Only the entry point to httpd, e.g. mod_proxy, mod_wl, mod_ws, mod_jk
> mod_cgi or whatnot should ever dechunk. The response body must pass
> through httpd without any protocol (chunking) or headers.
Ah, interesting.
Phusion Passenger is an Apache module itself:
http://www.modrails.com/
Passenger is the most used solution for production deployments in Ruby
on Rails nowadays.
So I understand from your reply that httpd is the only one resposible
for chunked responses, compressed or otherwise. Is that correct?
So a module like Passenger is expected to dechunk and pass the body
to httpd? As of this writing chunked responses are passing through
Apache if compression is not involved. Guess that works by luck, but
it is still not playing by the rules?
If whether the response has to be streamed or not is decided by the
application, which is the proper way to force httpd to stream, and
eventually also compress?
Use case is flushing the HEAD of HTML documents for dynamic pages
whose content generation is costly. So, while content is being
produced, the client is already fetching assets.
-- fxn
PS: Streaming is a new feature to be out with Rails 3.1, we are exploring
production setups that play well with compression.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 6/5/2011 12:31 PM, Xavier Noria wrote:
>
> I am testing this with Phusion Passenger and Unicorn.
>
> They are going to implement compression for chunked responses. That
> is, they are going to dechunk, compress, and chunk again, mod_deflate
> won't be involved for these responses.
httpd (conditionally) handles the chunking... the app generator's
chunking is never used. What *module* is installed in httpd? I'm
not familiar with the above.
Only the entry point to httpd, e.g. mod_proxy, mod_wl, mod_ws, mod_jk
mod_cgi or whatnot should ever dechunk. The response body must pass
through httpd without any protocol (chunking) or headers.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
On Sun, Jun 5, 2011 at 7:01 PM, William A. Rowe Jr. <wr...@rowe-clan.net> wrote:
> you need to be more specific.
>
> HTTP 2.x has a filtering schema which applies -protocol- filters
> after all -content-. Modules are presumed to generate content
> unless they manipulate the filter stack.
>
> mod_proxy dechunks the backend response (it's entirely possible
> for httpd to accept chunking, and the proxy client to not accept
> it as HTTP/1.0), since T-E is a hop-by-hop header.
>
> So whatever *app server* you have that is chunking and not being
> dechunked is an issue with the module you have receiving the
> chunk response in httpd. Chunking is not part of the CGI spec,
> therefore it's invalid input to mod_cgi[d]. It's completely
> legitimate for mod_proxy_http. Tomcat has no such issues that
> I'm aware of.
>
> So what app server are you talking about?
I am testing this with Phusion Passenger and Unicorn.
They are going to implement compression for chunked responses. That
is, they are going to dechunk, compress, and chunk again, mod_deflate
won't be involved for these responses.
Is that what you'd expect to be needed?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 6/5/2011 11:47 AM, Xavier Noria wrote:
>
> Also, it is clear that mod_deflate does not understand chunked
> encoding coming from the app server. It compresses the payload.
Xavier,
you need to be more specific.
HTTP 2.x has a filtering schema which applies -protocol- filters
after all -content-. Modules are presumed to generate content
unless they manipulate the filter stack.
mod_proxy dechunks the backend response (it's entirely possible
for httpd to accept chunking, and the proxy client to not accept
it as HTTP/1.0), since T-E is a hop-by-hop header.
So whatever *app server* you have that is chunking and not being
dechunked is an issue with the module you have receiving the
chunk response in httpd. Chunking is not part of the CGI spec,
therefore it's invalid input to mod_cgi[d]. It's completely
legitimate for mod_proxy_http. Tomcat has no such issues that
I'm aware of.
So what app server are you talking about?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_deflate and chunked encoding
Posted by Xavier Noria <fx...@hashref.com>.
I think that's definitely not correct. Browsers do inflate and process
the HTML on the fly, they do not wait for the entire payload.
Chrome seems to have a buffer of 256 bytes, and Firefox has none.
I have used this server for testing this:
https://gist.github.com/1009108
and monitored when foo.css was requested on port 3001.
Also, it is clear that mod_deflate does not understand chunked
encoding coming from the app server. It compresses the payload.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org