You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Filipe David Manana <fd...@gmail.com> on 2010/03/23 01:09:44 UTC

Re: standalone attachments and content-encoding header

On Thu, Mar 18, 2010 at 12:19 PM, Brian Candler <B....@pobox.com> wrote:

> On Wed, Mar 17, 2010 at 03:52:36PM -0700, Randall Leeds wrote:
> > It seems like couch should probably store content-encoding as metadata
> > on the attachment (like content-type) and pass these attachments
> > straight to disk and serve them up with the appropriate
> > content-encoding
>
> What happens if another client requests the same document but doesn't
> understand that particular Content-Encoding?  Couchdb would then have to
> refuse to *serve* the document out to the second client, if its
> Accept-Encoding: list didn't include the encoding stored to disk.
>

> Or couchdb would have to transcode on demand, potentially twice:
> deflate->identity->gzip for example.
>

Somehow I feel that transcoding is a bit unnatural and consumes too much cpu
for large attachments.


> > OR reject the original upload as unsupported.
>

This would definitely be the simplest approach, by accepting only the "gzip"
content encoding.

By accepting an already gzip encoded attachment (through the PUT standalone
attachment API) we risk not storing the real length of the attachment
(length of the uncompressed form) in the attachment's metadata, and only the
length of the attachment's encoded form.
That would happen only if the attachment is streamed with a chunked transfer
encoding request (that i.e. no content-length header in the upload request).

Also, supporting multiple content-encodings would imply doing some changes
to the code introduced in ticket 639 (applied to trunk only) which adds a
new doc query parameter ?att_gzip_length=true. (replacing it with 2
parameters such as encoding=<encoding_name> and encoded_length=123).

I am more inclined for supporting only the gzip encoding for attachments,
but this should be a community decision, therefore I include this reply to
the dev mailing list.


> I think it should do this anyway for JSON document bodies. Couchdb
> shouldn't
> attempt to JSON-parse the body if it has come in using an encoding that it
> doesn't understand.
>



-- 
Filipe David Manana,
fdmanana@gmail.com

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."

Re: standalone attachments and content-encoding header

Posted by J Chris Anderson <jc...@gmail.com>.
On Mar 23, 2010, at 6:40 AM, Filipe David Manana wrote:

> On Tue, Mar 23, 2010 at 12:36 AM, J Chris Anderson <jc...@gmail.com> wrote:
> 
>> So we should make sure our public api will accommodate patches to add
>> support for deflate and compress. This is more important than actually
>> supporting those encodings, for now.
>> 
>> 
> In that case, if nobody is against it, I'll do a patch that:
> 
> - removes the ?att_gzip_length=true doc query parameter
> - adds the doc query parameter ?att_encoding_info=true
> - adds two fields to an attachment stub, only when the doc is queried with
> the above parameter, that are "encoding" (currently the value will always be
> "gzip") and "encoded_length" (which replaces the "gzip_length" field).
> 
> This would imply also adding a new extra field to the record #att named
> "encoding" for e.g., or better yet, renaming #att.comp to #att.encoding
> (currently with only 2 possible values "identity" and "gzip"), which in turn
> implies changing the DB file format :(
> 
> Perhaps those 2 attachment stub properties should be always present? Without
> needing that doc query parameter "att_encoding_info=true"
> 

I think it'd be good to keep the file format unchanged (even if the names aren't 100% descriptive -- "comp" seems ok for now, if there are code comments added.)

It should also be possible to use pattern matching to treat comp = true as comp = "gzip".

Also, I don't think there is a rush on this stuff, just as long as we move in the proper direction.

Chris

> 
>> Chris
>> 
>>> --
>>> Filipe David Manana,
>>> fdmanana@gmail.com
>>> 
>>> "Reasonable men adapt themselves to the world.
>>> Unreasonable men adapt the world to themselves.
>>> That's why all progress depends on unreasonable men."
>> 
>> 
> 
> 
> -- 
> Filipe David Manana,
> fdmanana@gmail.com
> 
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."


Re: standalone attachments and content-encoding header

Posted by Filipe David Manana <fd...@gmail.com>.
On Tue, Mar 23, 2010 at 12:36 AM, J Chris Anderson <jc...@gmail.com> wrote:

> So we should make sure our public api will accommodate patches to add
> support for deflate and compress. This is more important than actually
> supporting those encodings, for now.
>
>
In that case, if nobody is against it, I'll do a patch that:

- removes the ?att_gzip_length=true doc query parameter
- adds the doc query parameter ?att_encoding_info=true
- adds two fields to an attachment stub, only when the doc is queried with
the above parameter, that are "encoding" (currently the value will always be
"gzip") and "encoded_length" (which replaces the "gzip_length" field).

This would imply also adding a new extra field to the record #att named
"encoding" for e.g., or better yet, renaming #att.comp to #att.encoding
(currently with only 2 possible values "identity" and "gzip"), which in turn
implies changing the DB file format :(

Perhaps those 2 attachment stub properties should be always present? Without
needing that doc query parameter "att_encoding_info=true"


> Chris
>
> > --
> > Filipe David Manana,
> > fdmanana@gmail.com
> >
> > "Reasonable men adapt themselves to the world.
> > Unreasonable men adapt the world to themselves.
> > That's why all progress depends on unreasonable men."
>
>


-- 
Filipe David Manana,
fdmanana@gmail.com

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."

Re: standalone attachments and content-encoding header

Posted by J Chris Anderson <jc...@gmail.com>.
On Mar 22, 2010, at 5:24 PM, Filipe David Manana wrote:

> On Tue, Mar 23, 2010 at 12:09 AM, Filipe David Manana <fd...@gmail.com>wrote:
> 
>> 
>> 
>> Also, supporting multiple content-encodings would imply doing some changes
>> to the code introduced in ticket 639 (applied to trunk only) which adds a
>> new doc query parameter ?att_gzip_length=true. (replacing it with 2
>> parameters such as encoding=<encoding_name> and encoded_length=123).
>> 
> 
> Here I meant:
> 
> 1) renaming doc query param "att_gzip_length" to something like
> "att_encoded_length"
> 2) replacing the attachment stub field "gzip_length" with the 2 fields
> "encoding" and "encoded_length"
> 
> Overall there are only 4 content encodings defined in the http rfc 2616:
> identity, gzip, deflate (zlib's default) and compress (Lempel-Ziv
> compression)
> 

I'd suggest that we don't worry supporting any of those we don't already support. But in the long run we'll end up supporting the full set of them (as well as all the transcoding uglies and everything else...)

So we should make sure our public api will accommodate patches to add support for deflate and compress. This is more important than actually supporting those encodings, for now.

Chris

> -- 
> Filipe David Manana,
> fdmanana@gmail.com
> 
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."


Re: standalone attachments and content-encoding header

Posted by Filipe David Manana <fd...@gmail.com>.
On Tue, Mar 23, 2010 at 12:09 AM, Filipe David Manana <fd...@gmail.com>wrote:

>
>
> Also, supporting multiple content-encodings would imply doing some changes
> to the code introduced in ticket 639 (applied to trunk only) which adds a
> new doc query parameter ?att_gzip_length=true. (replacing it with 2
> parameters such as encoding=<encoding_name> and encoded_length=123).
>

Here I meant:

1) renaming doc query param "att_gzip_length" to something like
"att_encoded_length"
2) replacing the attachment stub field "gzip_length" with the 2 fields
"encoding" and "encoded_length"

Overall there are only 4 content encodings defined in the http rfc 2616:
identity, gzip, deflate (zlib's default) and compress (Lempel-Ziv
compression)

-- 
Filipe David Manana,
fdmanana@gmail.com

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."

Re: standalone attachments and content-encoding header

Posted by Filipe David Manana <fd...@gmail.com>.
On Tue, Mar 23, 2010 at 12:09 AM, Filipe David Manana <fd...@gmail.com>wrote:

>
>
> Also, supporting multiple content-encodings would imply doing some changes
> to the code introduced in ticket 639 (applied to trunk only) which adds a
> new doc query parameter ?att_gzip_length=true. (replacing it with 2
> parameters such as encoding=<encoding_name> and encoded_length=123).
>

Here I meant:

1) renaming doc query param "att_gzip_length" to something like
"att_encoded_length"
2) replacing the attachment stub field "gzip_length" with the 2 fields
"encoding" and "encoded_length"

Overall there are only 4 content encodings defined in the http rfc 2616:
identity, gzip, deflate (zlib's default) and compress (Lempel-Ziv
compression)

-- 
Filipe David Manana,
fdmanana@gmail.com

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."

Re: standalone attachments and content-encoding header

Posted by Brian Candler <B....@pobox.com>.
On Tue, Mar 23, 2010 at 11:53:38AM +0000, Filipe David Manana wrote:
> If you upload attachment A in uncompressed form, the "length" field in the
> attachment stub will match the length of the attachment in uncompressed
> form. On the other hand, if you upload that same attachment in compressed
> form, that "length" field will match the length of the attachment in
> compressed form.

That would be OK if the first had no content_encoding field stored, and the
latter had content_encoding: "gzip".  In both cases the content_length would
be an accurate reflection of what was stored, in the form it was stored.

(It might be a bit odd that two clients could upload the 'same' document,
one encoded and one not, and you'd see them differently.  It depends whether
you think it's couchdb's job to normalise the documents, or just store what
it's given)

> Uncompressing the attachment just to calculate its identity length seems a
> bit heavy, no?

Well yes, but it's better than just being wrong(*) as couchdb is at the
moment.

A simpler solution would be for couchdb to reject all uploads which have any
Content-Encoding: header with a 415.  That's correct but brutal, and
asymmetrical with attachment downloading where a request with
"Accept-Encoding: gzip" can return you a "Content-Encoding: gzip".

If it's going to accept encodings on upload then it either needs to store
the content-encoding (and return it on a subsequent GET), or normalise to
identity form, which involves decompressing [if only to get the length] but
is otherwise simple and unsurprising.

If it's going to store the content-encoding then it needs to be decided how
this interacts with the transparent gzipping to disk which (usually) occurs
when uploading attachments.

One option is to reveal it: that is, the user could upload a document and
then subsequently see
  content_encoding: "gzip"
  content_length: <some smaller value>
That's an API change and might be surprising.

But if not, there might be two different ways a gzipped doc could be stored
on disk; either explicitly (uploaded with Content-Encoding: gzip) or
implicitly (uploaded as identity, couchdb decided to gzip it). Normalising
avoids this issue.

Incidentally, I don't think couchdb *always* gzips attachments. I remember
reading a long discussion on a ticket about heuristics to avoid
re-compressing anything already compressed, or inherently incompressible.

Also: the same issue arises if we decide to store content_md5 (which would
be very useful in determining if an attachment has changed).

B.

(*) As shown by:

$ ls -l /usr/share/file/magic
-rw-r--r-- 1 root root 544773 2009-05-13 18:35 /usr/share/file/magic
$ cat /usr/share/file/magic | gzip -9 > magic.gz
$ ls -l magic.gz 
-rw-r--r-- 1 brian brian 151706 2010-03-23 12:35 magic.gz
$ curl -X PUT -d'{}' http://127.0.0.1:5984/testdb/foo
{"ok":true,"id":"foo","rev":"1-967a00dff5e02add41819138abb3284d"}
$ curl -v -X PUT --data-binary @magic.gz -H "Content-Encoding: gzip" http://127.0.0.1:5984/testdb/foo/att?rev=1-967a00dff5e02add41819138abb3284d
* About to connect() to 127.0.0.1 port 5984 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
> PUT /testdb/foo/att?rev=1-967a00dff5e02add41819138abb3284d HTTP/1.1
> User-Agent: curl/7.19.5 (x86_64-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:5984
> Accept: */*
> Content-Encoding: gzip
> Content-Length: 151706
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< Server: CouchDB/0.11.0b912a3e00-git (Erlang OTP/R13B)
< Location: http://127.0.0.1:5984/testdb/foo/att
< Date: Tue, 23 Mar 2010 12:37:45 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 66
< Cache-Control: must-revalidate
< 
{"ok":true,"id":"foo","rev":"2-4a6c43c3c8b26c7fdbdc956c0d2477a7"}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0
$ curl http://127.0.0.1:5984/testdb/foo
{"_id":"foo","_rev":"2-4a6c43c3c8b26c7fdbdc956c0d2477a7","_attachments":{"att":{"content_type":"application/x-www-form-urlencoded","revpos":2,"length":151706,"stub":true}}}
$ curl -v http://127.0.0.1:5984/testdb/foo/att | wc -c
* About to connect() to 127.0.0.1 port 5984 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
> GET /testdb/foo/att HTTP/1.1
> User-Agent: curl/7.19.5 (x86_64-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:5984
> Accept: */*
> 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: CouchDB/0.11.0b912a3e00-git (Erlang OTP/R13B)
< ETag: "2-4a6c43c3c8b26c7fdbdc956c0d2477a7"
< Date: Tue, 23 Mar 2010 12:51:43 GMT
< Content-Type: application/x-www-form-urlencoded
< Cache-Control: must-revalidate
< 
{ [data not shown]
100  148k    0  148k    0     0  12.6M      0 --:--:-- --:--:-- --:--:-- 14.4M* Connection #0 to host 127.0.0.1 left intact

* Closing connection #0
151706

Re: standalone attachments and content-encoding header

Posted by Filipe David Manana <fd...@gmail.com>.
On Tue, Mar 23, 2010 at 9:05 AM, Brian Candler <B....@pobox.com> wrote:

> On Tue, Mar 23, 2010 at 12:09:44AM +0000, Filipe David Manana wrote:
> > By accepting an already gzip encoded attachment (through the PUT
> standalone
> > attachment API) we risk not storing the real length of the attachment
> > (length of the uncompressed form) in the attachment's metadata, and only
> the
> > length of the attachment's encoded form.
> > That would happen only if the attachment is streamed with a chunked
> transfer
> > encoding request (that i.e. no content-length header in the upload
> request).
>
> I thought content-length referred to the actual number of bytes transferred
> (i.e. the encoded form), not the original uncompressed object? So to get
> the
> "real" size you'd have to decompress anyway?
>

True. I guess I shouldn't reply anymore during late evening :)

The problem I see here is that not supplying, in an attachment stub, the
length of the attachment in uncompressed form breaks the existing API.
Currently the "length" field of an attachment stub is always the length of
the attachment in its identity (uncompressed) form.

If you upload attachment A in uncompressed form, the "length" field in the
attachment stub will match the length of the attachment in uncompressed
form. On the other hand, if you upload that same attachment in compressed
form, that "length" field will match the length of the attachment in
compressed form.

Uncompressing the attachment just to calculate its identity length seems a
bit heavy, no?




-- 
Filipe David Manana,
fdmanana@gmail.com

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."

Re: standalone attachments and content-encoding header

Posted by Simon Metson <si...@googlemail.com>.
Hi,

>> That would happen only if the attachment is streamed with a chunked  
>> transfer
>> encoding request (that i.e. no content-length header in the upload  
>> request).
>
> I thought content-length referred to the actual number of bytes  
> transferred
> (i.e. the encoded form), not the original uncompressed object? So to  
> get the
> "real" size you'd have to decompress anyway?

Yup: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

Re: standalone attachments and content-encoding header

Posted by Brian Candler <B....@pobox.com>.
On Tue, Mar 23, 2010 at 12:09:44AM +0000, Filipe David Manana wrote:
> By accepting an already gzip encoded attachment (through the PUT standalone
> attachment API) we risk not storing the real length of the attachment
> (length of the uncompressed form) in the attachment's metadata, and only the
> length of the attachment's encoded form.
> That would happen only if the attachment is streamed with a chunked transfer
> encoding request (that i.e. no content-length header in the upload request).

I thought content-length referred to the actual number of bytes transferred
(i.e. the encoded form), not the original uncompressed object? So to get the
"real" size you'd have to decompress anyway?