You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Louis Gerbarg <lg...@gmail.com> on 2009/01/31 07:02:37 UTC

Content-Encoding: gzip

I am using CouchDB as backend for some iPhone applications, and one of
the issues I have is that that the responses are verbose, which is an
issue over cellular. I decided to take a crack at gzip support, and I
have something working. A few of you might have seen my post about
this on the mochiweb list a few days ago, I think I have a solution
that will be more acceptable for mochiweb, so a significant chunk of
the code might not really be in CouchDB itself.

Having said that, I have posted a branch with the first cut on github.
It works for all GET requests. It also adds buffering to coalesce
small chunks together even when gzip is not being used. This is my
first erlang code, so it is definitely not not what I would consider
production quality.

In particular:

Some of the code is in the wrong place (like type negotiation)
It is not properly documented, since I have not read up on Erlang
documentation style
It could do better compression with a little more work (fully flushes
data and resets state when it could avoid that)

On the other hand, it appears to generally work and for my usage it
already is a pretty big win with no changes in our client app ;-) I
will be cleaning it up a bit, trying to make it acceptable to
mochiweb, perform better, etc. If anyone is interested you can grab
the current patchset from
<http://github.com/lgerbarg/couchdb/tree/gzip-support>.

Louis

Re: Content-Encoding: gzip

Posted by Louis Gerbarg <lg...@gmail.com>.
On Sat, Jan 31, 2009 at 1:52 AM, Chris Anderson <jc...@apache.org> wrote:
> On Fri, Jan 30, 2009 at 10:02 PM, Louis Gerbarg <lg...@gmail.com> wrote:
>> I am using CouchDB as backend for some iPhone applications, and one of
>> the issues I have is that that the responses are verbose, which is an
>> issue over cellular. I decided to take a crack at gzip support, and I
>> have something working. A few of you might have seen my post about
>> this on the mochiweb list a few days ago, I think I have a solution
>> that will be more acceptable for mochiweb, so a significant chunk of
>> the code might not really be in CouchDB itself.
>>
>
> I've been following the discussion on Mochiweb's list. Thanks for
> digging into this.
>
>> Having said that, I have posted a branch with the first cut on github.
>> It works for all GET requests. It also adds buffering to coalesce
>> small chunks together even when gzip is not being used. This is my
>> first erlang code, so it is definitely not not what I would consider
>> production quality.
>>
>
> I think we'd like to have this feature. There are some subtleties for
> Couch. One is that for attachments, we may like to receive them
> gzipped, store them and send them to clients, all without doing any
> reprocessing. This may not be related directly to your patch, but you
> should know about it, in case you see places where you could work it
> in.

Yeah, that had crossed my mind. Also, there are some attachments we
will not one to use streaming compression on (images, movies, zipped
files, etc). Right now I am doing it for basically everything, though
I will probably (for the time being) disable it on attachments until
we at least figure out how to deal with mimetypes.

>> In particular:
>>
>> Some of the code is in the wrong place (like type negotiation)
>> It is not properly documented, since I have not read up on Erlang
>> documentation style
>> It could do better compression with a little more work (fully flushes
>> data and resets state when it could avoid that)
>
> Do you feel up to polishing it off or are you hoping to find a
> collaborator to get it ready for prime time?

I should be fine to finish it off, but I will definitely want someone
to double check/code review my stuff. I have a lot of experience
dealing with http stacks, and spent many years writing a lot of
LISP/Scheme code, but I am completely new to Erlang and in some cases
I have been guessing how somethings work. Features like parameterized
namespaces are not necessarily well documented ;-) I will try to get
it into a form that that I feel is decent in the next few days then
solicit feedback on that.

>>
>> On the other hand, it appears to generally work and for my usage it
>> already is a pretty big win with no changes in our client app ;-) I
>> will be cleaning it up a bit, trying to make it acceptable to
>> mochiweb, perform better, etc. If anyone is interested you can grab
>> the current patchset from
>> <http://github.com/lgerbarg/couchdb/tree/gzip-support>.
>>
>
> Thanks again for digging into details.

No problem, thanks for providing a fantastic database platform for me to use.

Louis

Re: Content-Encoding: gzip

Posted by Chris Anderson <jc...@apache.org>.
On Fri, Jan 30, 2009 at 10:02 PM, Louis Gerbarg <lg...@gmail.com> wrote:
> I am using CouchDB as backend for some iPhone applications, and one of
> the issues I have is that that the responses are verbose, which is an
> issue over cellular. I decided to take a crack at gzip support, and I
> have something working. A few of you might have seen my post about
> this on the mochiweb list a few days ago, I think I have a solution
> that will be more acceptable for mochiweb, so a significant chunk of
> the code might not really be in CouchDB itself.
>

I've been following the discussion on Mochiweb's list. Thanks for
digging into this.

> Having said that, I have posted a branch with the first cut on github.
> It works for all GET requests. It also adds buffering to coalesce
> small chunks together even when gzip is not being used. This is my
> first erlang code, so it is definitely not not what I would consider
> production quality.
>

I think we'd like to have this feature. There are some subtleties for
Couch. One is that for attachments, we may like to receive them
gzipped, store them and send them to clients, all without doing any
reprocessing. This may not be related directly to your patch, but you
should know about it, in case you see places where you could work it
in.

> In particular:
>
> Some of the code is in the wrong place (like type negotiation)
> It is not properly documented, since I have not read up on Erlang
> documentation style
> It could do better compression with a little more work (fully flushes
> data and resets state when it could avoid that)

Do you feel up to polishing it off or are you hoping to find a
collaborator to get it ready for prime time?

>
> On the other hand, it appears to generally work and for my usage it
> already is a pretty big win with no changes in our client app ;-) I
> will be cleaning it up a bit, trying to make it acceptable to
> mochiweb, perform better, etc. If anyone is interested you can grab
> the current patchset from
> <http://github.com/lgerbarg/couchdb/tree/gzip-support>.
>

Thanks again for digging into details.



-- 
Chris Anderson
http://jchris.mfdz.com