You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Lieven Govaerts <lg...@apache.org> on 2012/11/13 08:41:53 UTC
Re: Double compression over HTTPS
On Fri, Oct 5, 2012 at 11:51 AM, Lieven Govaerts <lg...@apache.org> wrote:
> Hi,
>
>
>
> when OpenSSL is built with zlib, it will automatically compress all data
> sent over an SSL connection. You can see this in the initial handshake
> "Client Hello" and "Server Hello" where client and server agree on the
> compression mechanism to be used.
>
> If the data being sent or received is already compressed, OpenSSL will
> compress it a second time. This can happen when already compressed binaries
> like .gif or .zip are sent, or when the server uses gzip encoding for a http
> response. This can have impact on performance and memory usage. See Paul
> Querna's blog post about this topic in [1]. This also has been mentioned
> before on svn-dev by Justin in [2].
>
> Since OpenSSL 1.0 this automatic compression can be disabled at runtime.
>
> Compression by OpenSSL has some advantages and disadvantages:
> + OpenSSL will compress the full data stream, so for https that includes all
> headers + all small requests and responses which mod_deflate skips.
> + OpenSSL compression is stateful, it will not reset its dictionary between
> every response like gzip/deflate-encoding does, so it will reach better
> compression ratio when content of multiple consecutive requests or responses
> are similar within a 32KB window. Side note: I have done tests with using a
> preset dictionary for zlib for http(s) responses and found the difference
> can be up to 50% extra compression.
> - Where content is already compressed by the application layer (e.g. gzip
> encoding or transferring binary files), OpenSSL will compress these again.
>
>
> I have been doing some small-scale testing to see what difference this all
> makes. My test case was using svn to checkout a copy of the subversion trunk
> branch in the asf repository.
>
> I have tested 4 different scenario's:
> 1. As-is setup, OpenSSL compression enabled + gzip encoding enabled. (double
> compression)
> 2. OpenSSL compression disabled + gzip encoding enabled. (compression
> handled by the application)
> 3. OpenSSL compression disabled + gzip encoding disabled. (no compression at
> all)
> 4. OpenSSL compression enabled + gzip encoding disabled (compression handled
> by OpenSSL)
>
> I found this particular scenario too small to see a measurable difference in
> memory or cpu usage, although this is interesting to test further.
>
> Difference in total times are more interesting:
> | bytes read | bytes written | total time
> 1: | 17.50MB | 233-284KB | 59s
> 2: | 18.67MB | 2.13-2.43MB | 1m9s-1m18s
> 3: | 50.35MB | 2.34MB | 103s-108s
> 4: | 15.27MB | 235-260KB | 50s-56s
>
> You can see from the above reasoning and my test results that it would be
> beneficial to disable gzip encoding when using https if OpenSSL was built
> with zlib. However, in the scenario where large compressed binary files are
> stored in a svn repository, I suppose disabling both OpenSSL compression and
> gzip encoding will provide the best results.
>
> Given the above I propose the following:
> - Add an option in serf to disable OpenSSL compression
> - Add a function in serf to check if compression is enabled in OpenSSL.
> - In Subversion, don't ask for gzip encoding when working over https with
> compression.
> - In Subversion, if the config option "http-compression" is set to "no",
> disable both OpenSSL compression and gzip encoding.
>
> Which makes scenario 4 the default, and the user can select for scenario 3
> with the "http-compression" option.
>
> Patch to disable OpenSSL compression in serf is attached.
>
> Suggestions? Objections?
This topic seems to be coming up in issue #3980 - serf increases
server load, so I'd like to clear that none of the proposed changes
have been implemented.
The reason is a confirmed security issue with openssl compression:
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4929
https://issues.apache.org/bugzilla/show_bug.cgi?id=53219
The proposed approach here is to disable SSL compression completely,
so in terms of the above that leaves us with scenario 2.
Serf doesn't have an option currently to disable SSL compression from
the client side. I plan to add it in the next version.
Lieven
> [1]: http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/
> [2]: http://svn.haxx.se/dev/archive-2011-05/0362.shtml
>
Re: [serf-dev] Re: Double compression over HTTPS
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Tue, Nov 13, 2012 at 2:41 AM, Lieven Govaerts <lg...@apache.org> wrote:
> The proposed approach here is to disable SSL compression completely,
> so in terms of the above that leaves us with scenario 2.
>
> Serf doesn't have an option currently to disable SSL compression from
> the client side. I plan to add it in the next version.
+1. -- justin
Re: [serf-dev] Re: Double compression over HTTPS
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On Tue, Nov 13, 2012 at 11:41 AM, Lieven Govaerts <lg...@apache.org> wrote:
> On Fri, Oct 5, 2012 at 11:51 AM, Lieven Govaerts <lg...@apache.org> wrote:
>> Hi,
>> when OpenSSL is built with zlib, it will automatically compress all data
>> sent over an SSL connection. You can see this in the initial handshake
>> "Client Hello" and "Server Hello" where client and server agree on the
>> compression mechanism to be used.
>>
>> If the data being sent or received is already compressed, OpenSSL will
>> compress it a second time. This can happen when already compressed binaries
>> like .gif or .zip are sent, or when the server uses gzip encoding for a http
>> response. This can have impact on performance and memory usage. See Paul
>> Querna's blog post about this topic in [1]. This also has been mentioned
>> before on svn-dev by Justin in [2].
>>
>> Since OpenSSL 1.0 this automatic compression can be disabled at runtime.
>>
>> Compression by OpenSSL has some advantages and disadvantages:
>> + OpenSSL will compress the full data stream, so for https that includes all
>> headers + all small requests and responses which mod_deflate skips.
>> + OpenSSL compression is stateful, it will not reset its dictionary between
>> every response like gzip/deflate-encoding does, so it will reach better
>> compression ratio when content of multiple consecutive requests or responses
>> are similar within a 32KB window. Side note: I have done tests with using a
>> preset dictionary for zlib for http(s) responses and found the difference
>> can be up to 50% extra compression.
>> - Where content is already compressed by the application layer (e.g. gzip
>> encoding or transferring binary files), OpenSSL will compress these again.
>>
>>
>> I have been doing some small-scale testing to see what difference this all
>> makes. My test case was using svn to checkout a copy of the subversion trunk
>> branch in the asf repository.
>>
>> I have tested 4 different scenario's:
>> 1. As-is setup, OpenSSL compression enabled + gzip encoding enabled. (double
>> compression)
>> 2. OpenSSL compression disabled + gzip encoding enabled. (compression
>> handled by the application)
>> 3. OpenSSL compression disabled + gzip encoding disabled. (no compression at
>> all)
>> 4. OpenSSL compression enabled + gzip encoding disabled (compression handled
>> by OpenSSL)
>>
>> I found this particular scenario too small to see a measurable difference in
>> memory or cpu usage, although this is interesting to test further.
>>
>> Difference in total times are more interesting:
>> | bytes read | bytes written | total time
>> 1: | 17.50MB | 233-284KB | 59s
>> 2: | 18.67MB | 2.13-2.43MB | 1m9s-1m18s
>> 3: | 50.35MB | 2.34MB | 103s-108s
>> 4: | 15.27MB | 235-260KB | 50s-56s
>>
>> You can see from the above reasoning and my test results that it would be
>> beneficial to disable gzip encoding when using https if OpenSSL was built
>> with zlib. However, in the scenario where large compressed binary files are
>> stored in a svn repository, I suppose disabling both OpenSSL compression and
>> gzip encoding will provide the best results.
>>
>> Given the above I propose the following:
>> - Add an option in serf to disable OpenSSL compression
>> - Add a function in serf to check if compression is enabled in OpenSSL.
>> - In Subversion, don't ask for gzip encoding when working over https with
>> compression.
>> - In Subversion, if the config option "http-compression" is set to "no",
>> disable both OpenSSL compression and gzip encoding.
>>
>> Which makes scenario 4 the default, and the user can select for scenario 3
>> with the "http-compression" option.
>>
>> Patch to disable OpenSSL compression in serf is attached.
>>
>> Suggestions? Objections?
>
> This topic seems to be coming up in issue #3980 - serf increases
> server load, so I'd like to clear that none of the proposed changes
> have been implemented.
> The reason is a confirmed security issue with openssl compression:
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4929
> https://issues.apache.org/bugzilla/show_bug.cgi?id=53219
>
> The proposed approach here is to disable SSL compression completely,
> so in terms of the above that leaves us with scenario 2.
>
> Serf doesn't have an option currently to disable SSL compression from
> the client side. I plan to add it in the next version.
>
>From my testing OpenSSL compression is also very slow when used for
local area connections.
--
Ivan Zhakov
Re: [serf-dev] Re: Double compression over HTTPS
Posted by Lieven Govaerts <li...@gmail.com>.
On Thu, Nov 15, 2012 at 1:01 AM, Greg Stein <gs...@gmail.com> wrote:
>
> On Nov 14, 2012 1:44 AM, "Lieven Govaerts" <lg...@apache.org> wrote:
>>...
>
>
>> Implemented in serf r1692, with an API for applications to enable SSL
>> compression. Which to be clear, I don't propose svn to use.
>
> This API would go into a 1.2 and 2.0, but if nobody is asking for it, then
> I'd suggest we don't spin those up yet.
>
Agreed on not making a release only for this change, but I do plan to
prepare a 1.2 release pretty soon, when the issue fixing work in
progress has been finalized.
Lieven
Re: [serf-dev] Re: Double compression over HTTPS
Posted by Greg Stein <gs...@gmail.com>.
On Nov 14, 2012 1:44 AM, "Lieven Govaerts" <lg...@apache.org> wrote:
>...
> Implemented in serf r1692, with an API for applications to enable SSL
> compression. Which to be clear, I don't propose svn to use.
This API would go into a 1.2 and 2.0, but if nobody is asking for it, then
I'd suggest we don't spin those up yet.
Cheers,
-g
Re: Double compression over HTTPS
Posted by Lieven Govaerts <lg...@apache.org>.
On Tue, Nov 13, 2012 at 8:41 AM, Lieven Govaerts <lg...@apache.org> wrote:
> On Fri, Oct 5, 2012 at 11:51 AM, Lieven Govaerts <lg...@apache.org> wrote:
>> Hi,
>>
>>
>>
>> when OpenSSL is built with zlib, it will automatically compress all data
>> sent over an SSL connection. You can see this in the initial handshake
>> "Client Hello" and "Server Hello" where client and server agree on the
>> compression mechanism to be used.
>>
>> If the data being sent or received is already compressed, OpenSSL will
>> compress it a second time. This can happen when already compressed binaries
>> like .gif or .zip are sent, or when the server uses gzip encoding for a http
>> response. This can have impact on performance and memory usage. See Paul
>> Querna's blog post about this topic in [1]. This also has been mentioned
>> before on svn-dev by Justin in [2].
>>
>> Since OpenSSL 1.0 this automatic compression can be disabled at runtime.
>>
>> Compression by OpenSSL has some advantages and disadvantages:
>> + OpenSSL will compress the full data stream, so for https that includes all
>> headers + all small requests and responses which mod_deflate skips.
>> + OpenSSL compression is stateful, it will not reset its dictionary between
>> every response like gzip/deflate-encoding does, so it will reach better
>> compression ratio when content of multiple consecutive requests or responses
>> are similar within a 32KB window. Side note: I have done tests with using a
>> preset dictionary for zlib for http(s) responses and found the difference
>> can be up to 50% extra compression.
>> - Where content is already compressed by the application layer (e.g. gzip
>> encoding or transferring binary files), OpenSSL will compress these again.
>>
>>
>> I have been doing some small-scale testing to see what difference this all
>> makes. My test case was using svn to checkout a copy of the subversion trunk
>> branch in the asf repository.
>>
>> I have tested 4 different scenario's:
>> 1. As-is setup, OpenSSL compression enabled + gzip encoding enabled. (double
>> compression)
>> 2. OpenSSL compression disabled + gzip encoding enabled. (compression
>> handled by the application)
>> 3. OpenSSL compression disabled + gzip encoding disabled. (no compression at
>> all)
>> 4. OpenSSL compression enabled + gzip encoding disabled (compression handled
>> by OpenSSL)
>>
>> I found this particular scenario too small to see a measurable difference in
>> memory or cpu usage, although this is interesting to test further.
>>
>> Difference in total times are more interesting:
>> | bytes read | bytes written | total time
>> 1: | 17.50MB | 233-284KB | 59s
>> 2: | 18.67MB | 2.13-2.43MB | 1m9s-1m18s
>> 3: | 50.35MB | 2.34MB | 103s-108s
>> 4: | 15.27MB | 235-260KB | 50s-56s
>>
>> You can see from the above reasoning and my test results that it would be
>> beneficial to disable gzip encoding when using https if OpenSSL was built
>> with zlib. However, in the scenario where large compressed binary files are
>> stored in a svn repository, I suppose disabling both OpenSSL compression and
>> gzip encoding will provide the best results.
>>
>> Given the above I propose the following:
>> - Add an option in serf to disable OpenSSL compression
>> - Add a function in serf to check if compression is enabled in OpenSSL.
>> - In Subversion, don't ask for gzip encoding when working over https with
>> compression.
>> - In Subversion, if the config option "http-compression" is set to "no",
>> disable both OpenSSL compression and gzip encoding.
>>
>> Which makes scenario 4 the default, and the user can select for scenario 3
>> with the "http-compression" option.
>>
>> Patch to disable OpenSSL compression in serf is attached.
>>
>> Suggestions? Objections?
>
> This topic seems to be coming up in issue #3980 - serf increases
> server load, so I'd like to clear that none of the proposed changes
> have been implemented.
> The reason is a confirmed security issue with openssl compression:
> http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4929
> https://issues.apache.org/bugzilla/show_bug.cgi?id=53219
>
> The proposed approach here is to disable SSL compression completely,
> so in terms of the above that leaves us with scenario 2.
>
> Serf doesn't have an option currently to disable SSL compression from
> the client side. I plan to add it in the next version.
>
Implemented in serf r1692, with an API for applications to enable SSL
compression. Which to be clear, I don't propose svn to use.
Lieven
>> [1]: http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/
>> [2]: http://svn.haxx.se/dev/archive-2011-05/0362.shtml
>>