mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

[Alex Stapleton <al...@prol.etari.at>]

(This has been cross posted to users@. I apologise if this mail isn't
relevant to the dev list.)

First some background. We use Apache HTTPD 2.0 over a high-latency,
high packet loss GPRS WAN. The cost per byte is tangible. We use SSL.
We also use Transfer-Encoding: chunked sometimes. This is a machine
monitoring application. We are using iframe streaming to push real
time data to operators browsers.

I have noticed after much faffing around with wireshark that httpd
will transmit 3 Application Data fragments for each chunk in a chunked
HTTP stream. The fragments are the HTTP chunk size, the chunk data,
and then a CRLF. All fragments in an SSL session are rounded to the
length of the ciphers block size. This results in the 4+2 bytes for
the chunk frame ending up being 64 bytes of data over TCP. A waste of
58 bytes per chunk in this case.

I'm not aware of any reason for this to behave specifically in this
way and clearly combining the entire chunk into a single SSL fragment
would provide a significant bandwidth saving for HTTP streaming
applications if not more mainstream ones.

I've done a fair amount of poking through the httpd source today to
try and isolate this but this is my first foray into the depths of
httpd so I've not got far in the direction of a solution. I have
identified that this 'problem' is due to the way the chunk_filter
function adds buckets into the brigade which end up getting turned
into their own fragments by mod_ssl. Creating a new bucket which has
the extra data wrapped around it would presumably be far too
inefficient for a general solution. I was considering using the FLUSH
bucket type but am not fully aware of how it's used by the various
filters.

I'm not sure what the ideal way is to go about fixing this, or if it's
even in fact an actual source code level problem rather than a
configuration one, hence why this is posted to users for now.

I can provide text dumps from tshark if that would be more
illuminating of the patterns I'm seeing.

Alex Stapleton

Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

[Alex Stapleton <al...@gmail.com>]

2009/8/21 Alex Stapleton <al...@gmail.com>:
> 2009/8/20 Roy T. Fielding <fi...@gbiv.com>:
>> On Aug 20, 2009, at 2:01 AM, Alex Stapleton wrote:
>>>
>>> 2009/8/19 Roy T. Fielding <fi...@gbiv.com>:
>>>>
>>>> On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote:
>>>>
>>>>> (This has been cross posted to users@. I apologise if this mail isn't
>>>>> relevant to the dev list.)
>>>>>
>>>>> First some background. We use Apache HTTPD 2.0 over a high-latency,
>>>>> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL.
>>>>> We also use Transfer-Encoding: chunked sometimes. This is a machine
>>>>> monitoring application. We are using iframe streaming to push real
>>>>> time data to operators browsers.
>>>>>
>>>>> I have noticed after much faffing around with wireshark that httpd
>>>>> will transmit 3 Application Data fragments for each chunk in a chunked
>>>>> HTTP stream. The fragments are the HTTP chunk size, the chunk data,
>>>>> and then a CRLF. All fragments in an SSL session are rounded to the
>>>>> length of the ciphers block size. This results in the 4+2 bytes for
>>>>> the chunk frame ending up being 64 bytes of data over TCP. A waste of
>>>>> 58 bytes per chunk in this case.
>>>>
>>>> That's odd -- we don't do this with non-SSL writes (last I checked).
>>>> Perhaps we are relying on a TCP cork for the non-SSL case?
>>>> What is your operating system and platform?
>>>
>>> I initially discovered this issue on a fairly old Ubuntu 6 machine
>>> running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X
>>> 10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also
>>> point out that I am using PHP 5.2.9 to generate the chunked data. If
>>> there's a way of doing chunked output using "plain" Apache let me know
>>> and I can test that too if needed.
>>
>> Er, have you tested it with 2.2.x?  The likelihood of us making any
>> non-security changes to the 2.0.x branch is extremely small.
>
> I have tested with 2.2.11 from MacPorts on my iMac and it also
> exhibits this behaviour. I'll try and do a 2.2.13 build to test with
> :)

Confirmed on 2.2.13.

>> Apache does chunked output by default if no content-length is
>> provided and the client says it is HTTP/1.1.
>>
>> ....Roy
>>
>>
>

Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

[Alex Stapleton <al...@gmail.com>]

2009/8/20 Roy T. Fielding <fi...@gbiv.com>:
> On Aug 20, 2009, at 2:01 AM, Alex Stapleton wrote:
>>
>> 2009/8/19 Roy T. Fielding <fi...@gbiv.com>:
>>>
>>> On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote:
>>>
>>>> (This has been cross posted to users@. I apologise if this mail isn't
>>>> relevant to the dev list.)
>>>>
>>>> First some background. We use Apache HTTPD 2.0 over a high-latency,
>>>> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL.
>>>> We also use Transfer-Encoding: chunked sometimes. This is a machine
>>>> monitoring application. We are using iframe streaming to push real
>>>> time data to operators browsers.
>>>>
>>>> I have noticed after much faffing around with wireshark that httpd
>>>> will transmit 3 Application Data fragments for each chunk in a chunked
>>>> HTTP stream. The fragments are the HTTP chunk size, the chunk data,
>>>> and then a CRLF. All fragments in an SSL session are rounded to the
>>>> length of the ciphers block size. This results in the 4+2 bytes for
>>>> the chunk frame ending up being 64 bytes of data over TCP. A waste of
>>>> 58 bytes per chunk in this case.
>>>
>>> That's odd -- we don't do this with non-SSL writes (last I checked).
>>> Perhaps we are relying on a TCP cork for the non-SSL case?
>>> What is your operating system and platform?
>>
>> I initially discovered this issue on a fairly old Ubuntu 6 machine
>> running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X
>> 10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also
>> point out that I am using PHP 5.2.9 to generate the chunked data. If
>> there's a way of doing chunked output using "plain" Apache let me know
>> and I can test that too if needed.
>
> Er, have you tested it with 2.2.x?  The likelihood of us making any
> non-security changes to the 2.0.x branch is extremely small.

I have tested with 2.2.11 from MacPorts on my iMac and it also
exhibits this behaviour. I'll try and do a 2.2.13 build to test with
:)

> Apache does chunked output by default if no content-length is
> provided and the client says it is HTTP/1.1.
>
> ....Roy
>
>

Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Roy T. Fielding wrote:
> 
> Apache does chunked output by default if no content-length is
> provided and the client says it is HTTP/1.1.

And further processes chunked input; it's not up to PHP or other
user agents to process this further, it's on the webserver to do so.

If you want to [de]chunk your own data, use mod_proxy_http.

Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

["Roy T. Fielding" <fi...@gbiv.com>]

On Aug 20, 2009, at 2:01 AM, Alex Stapleton wrote:
> 2009/8/19 Roy T. Fielding <fi...@gbiv.com>:
>> On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote:
>>
>>> (This has been cross posted to users@. I apologise if this mail  
>>> isn't
>>> relevant to the dev list.)
>>>
>>> First some background. We use Apache HTTPD 2.0 over a high-latency,
>>> high packet loss GPRS WAN. The cost per byte is tangible. We use  
>>> SSL.
>>> We also use Transfer-Encoding: chunked sometimes. This is a machine
>>> monitoring application. We are using iframe streaming to push real
>>> time data to operators browsers.
>>>
>>> I have noticed after much faffing around with wireshark that httpd
>>> will transmit 3 Application Data fragments for each chunk in a  
>>> chunked
>>> HTTP stream. The fragments are the HTTP chunk size, the chunk data,
>>> and then a CRLF. All fragments in an SSL session are rounded to the
>>> length of the ciphers block size. This results in the 4+2 bytes for
>>> the chunk frame ending up being 64 bytes of data over TCP. A  
>>> waste of
>>> 58 bytes per chunk in this case.
>>
>> That's odd -- we don't do this with non-SSL writes (last I checked).
>> Perhaps we are relying on a TCP cork for the non-SSL case?
>> What is your operating system and platform?
>
> I initially discovered this issue on a fairly old Ubuntu 6 machine
> running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X
> 10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also
> point out that I am using PHP 5.2.9 to generate the chunked data. If
> there's a way of doing chunked output using "plain" Apache let me know
> and I can test that too if needed.

Er, have you tested it with 2.2.x?  The likelihood of us making any
non-security changes to the 2.0.x branch is extremely small.

Apache does chunked output by default if no content-length is
provided and the client says it is HTTP/1.1.

....Roy

Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

[Alex Stapleton <al...@gmail.com>]

2009/8/19 Roy T. Fielding <fi...@gbiv.com>:
> On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote:
>
>> (This has been cross posted to users@. I apologise if this mail isn't
>> relevant to the dev list.)
>>
>> First some background. We use Apache HTTPD 2.0 over a high-latency,
>> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL.
>> We also use Transfer-Encoding: chunked sometimes. This is a machine
>> monitoring application. We are using iframe streaming to push real
>> time data to operators browsers.
>>
>> I have noticed after much faffing around with wireshark that httpd
>> will transmit 3 Application Data fragments for each chunk in a chunked
>> HTTP stream. The fragments are the HTTP chunk size, the chunk data,
>> and then a CRLF. All fragments in an SSL session are rounded to the
>> length of the ciphers block size. This results in the 4+2 bytes for
>> the chunk frame ending up being 64 bytes of data over TCP. A waste of
>> 58 bytes per chunk in this case.
>
> That's odd -- we don't do this with non-SSL writes (last I checked).
> Perhaps we are relying on a TCP cork for the non-SSL case?
> What is your operating system and platform?

I initially discovered this issue on a fairly old Ubuntu 6 machine
running httpd 2.0.55-4ubuntu4.1. I have since recreated it on my OS X
10.5 iMac using httpd 2.0.64 from MacPorts. I suppose I should also
point out that I am using PHP 5.2.9 to generate the chunked data. If
there's a way of doing chunked output using "plain" Apache let me know
and I can test that too if needed.

>> I'm not aware of any reason for this to behave specifically in this
>> way and clearly combining the entire chunk into a single SSL fragment
>> would provide a significant bandwidth saving for HTTP streaming
>> applications if not more mainstream ones.
>>
>> I've done a fair amount of poking through the httpd source today to
>> try and isolate this but this is my first foray into the depths of
>> httpd so I've not got far in the direction of a solution. I have
>> identified that this 'problem' is due to the way the chunk_filter
>> function adds buckets into the brigade which end up getting turned
>> into their own fragments by mod_ssl. Creating a new bucket which has
>> the extra data wrapped around it would presumably be far too
>> inefficient for a general solution. I was considering using the FLUSH
>> bucket type but am not fully aware of how it's used by the various
>> filters.
>
> It should not be necessary to have multiple buckets -- they should
> be written using a vector and not result in separate packets. However,
> this may be limited by the SSL library's write interface.
> FLUSH is the opposite of what you want.  We should either be doing
> the equivalent of a writev on SSL or add a buffering filter.
>
>> I'm not sure what the ideal way is to go about fixing this, or if it's
>> even in fact an actual source code level problem rather than a
>> configuration one, hence why this is posted to users for now.
>>
>> I can provide text dumps from tshark if that would be more
>> illuminating of the patterns I'm seeing.
>>
>> Alex Stapleton
>
> Platform details would be helpful -- you've narrowed the cause well
> enough that I doubt the text dumps would help.
>
> ....Roy
>

Re: mod_ssl and Transfer-Encoding: chunked wastes ~58 bytes per chunk.

["Roy T. Fielding" <fi...@gbiv.com>]

On Aug 19, 2009, at 9:53 AM, Alex Stapleton wrote:

> (This has been cross posted to users@. I apologise if this mail isn't
> relevant to the dev list.)
>
> First some background. We use Apache HTTPD 2.0 over a high-latency,
> high packet loss GPRS WAN. The cost per byte is tangible. We use SSL.
> We also use Transfer-Encoding: chunked sometimes. This is a machine
> monitoring application. We are using iframe streaming to push real
> time data to operators browsers.
>
> I have noticed after much faffing around with wireshark that httpd
> will transmit 3 Application Data fragments for each chunk in a chunked
> HTTP stream. The fragments are the HTTP chunk size, the chunk data,
> and then a CRLF. All fragments in an SSL session are rounded to the
> length of the ciphers block size. This results in the 4+2 bytes for
> the chunk frame ending up being 64 bytes of data over TCP. A waste of
> 58 bytes per chunk in this case.

That's odd -- we don't do this with non-SSL writes (last I checked).
Perhaps we are relying on a TCP cork for the non-SSL case?
What is your operating system and platform?

> I'm not aware of any reason for this to behave specifically in this
> way and clearly combining the entire chunk into a single SSL fragment
> would provide a significant bandwidth saving for HTTP streaming
> applications if not more mainstream ones.
>
> I've done a fair amount of poking through the httpd source today to
> try and isolate this but this is my first foray into the depths of
> httpd so I've not got far in the direction of a solution. I have
> identified that this 'problem' is due to the way the chunk_filter
> function adds buckets into the brigade which end up getting turned
> into their own fragments by mod_ssl. Creating a new bucket which has
> the extra data wrapped around it would presumably be far too
> inefficient for a general solution. I was considering using the FLUSH
> bucket type but am not fully aware of how it's used by the various
> filters.

It should not be necessary to have multiple buckets -- they should
be written using a vector and not result in separate packets. However,
this may be limited by the SSL library's write interface.
FLUSH is the opposite of what you want.  We should either be doing
the equivalent of a writev on SSL or add a buffering filter.

> I'm not sure what the ideal way is to go about fixing this, or if it's
> even in fact an actual source code level problem rather than a
> configuration one, hence why this is posted to users for now.
>
> I can provide text dumps from tshark if that would be more
> illuminating of the patterns I'm seeing.
>
> Alex Stapleton

Platform details would be helpful -- you've narrowed the cause well
enough that I doubt the text dumps would help.

....Roy