You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Steven Ringo <go...@stevenringo.com> on 2012/01/19 02:52:34 UTC
Attachments and replication
Hi all,
Assuming I have a document with an attachment and some fields, and that
this document has been replicated between source and target servers.
{
"_id": "551ffd9dc43cb73c257628d8a31e9e1f",
"_rev": "4-387d5d23b62e398421bddb10e16e740f",
"foo": "bar",
"baz": "quux",
"published": false,
"_attachments": {
"photo.jpg": {
"content_type": "image/jpeg",
"revpos": 2,
"digest": "md5-iOKjYGKWzEmEtMaqmYCtlA==",
"length": 3622469,
"stub": true
}
}
}
Let's say I change one of the fields on the source server; say "foo":
"bar" to "foo": "bar2". I then do a pull replication.
Sniffing the packets, it seems to me the entire document is replicated,
including attachments. I was under the impression couchdb would be able
to replicate the document independently of the attachments.
In other words couchdb would transfer just the "text/json" portion of
the document and not the attachment, i.e. less than ~1 kbyte vs. a few
megebytes in this case.
My experience seems to contradict my assumption. It looks like the
digest and revpos fields are used for this, as mentioned here:
http://stackoverflow.com/questions/1825117/what-is-the-revpos-value-used-for-in-couchdb-attachments
Is there something I am missing or perhaps a configuration option I am
not setting properly?
Thanks.
Re: Attachments and replication
Posted by Jens Alfke <je...@couchbase.com>.
What version of CouchDB? Filipe's new replicator should not be copying attachments unnecessarily.
--Jens [via iPhone]
On Jan 18, 2012, at 5:53 PM, "Steven Ringo" <go...@stevenringo.com> wrote:
> Hi all,
>
> Assuming I have a document with an attachment and some fields, and that
> this document has been replicated between source and target servers.
>
> {
> "_id": "551ffd9dc43cb73c257628d8a31e9e1f",
> "_rev": "4-387d5d23b62e398421bddb10e16e740f",
> "foo": "bar",
> "baz": "quux",
> "published": false,
> "_attachments": {
> "photo.jpg": {
> "content_type": "image/jpeg",
> "revpos": 2,
> "digest": "md5-iOKjYGKWzEmEtMaqmYCtlA==",
> "length": 3622469,
> "stub": true
> }
> }
> }
>
> Let's say I change one of the fields on the source server; say "foo":
> "bar" to "foo": "bar2". I then do a pull replication.
>
> Sniffing the packets, it seems to me the entire document is replicated,
> including attachments. I was under the impression couchdb would be able
> to replicate the document independently of the attachments.
>
> In other words couchdb would transfer just the "text/json" portion of
> the document and not the attachment, i.e. less than ~1 kbyte vs. a few
> megebytes in this case.
>
> My experience seems to contradict my assumption. It looks like the
> digest and revpos fields are used for this, as mentioned here:
> http://stackoverflow.com/questions/1825117/what-is-the-revpos-value-used-for-in-couchdb-attachments
>
> Is there something I am missing or perhaps a configuration option I am
> not setting properly?
>
> Thanks.
>
>
>
>
Re: Attachments and replication
Posted by Robert Newson <rn...@apache.org>.
The new replicator arrives with 1.2 and, as I understand it, is more
careful about not replicating attachments that haven't changed.
The earlier suggestion to use external files is bizarre, since they
will require management, cause other issues, will not be replicated,
will not be deleted, etc. It would, as you rightly saw, defeat most of
the value of CouchDB's replication feature. A future version of
CouchDB might allow the placement of attachment data into external
files but would manage them as you would expect.
B.
On 19 January 2012 03:40, Steven Ringo <go...@stevenringo.com> wrote:
> Jens,
>
> In terms of what you describe, I am definitely getting undesired behaviour.
>
> I am using CouchDB 1.1.1 on Mac OS replicating from the same instance, as
> well as IrisCouch (1.1.1 on Ubuntu, I think).
>
> Ultimately need this to work on Mobile couchbase, but experimenting with
> replication locally first.
>
> I ran:
>
> $ curl -H 'Content-Type: application/json' -X POST
> http://localhost:5984/_replicator -d '{"source":
> "http://127.0.0.1:5984/dslcollection", "target":
> "http://127.0.0.1:5984/dslcollection-ios"}'
> {"ok":true,"id":"f36d2bb4d5bdfd3813eb706130001632","rev":"1-c414a5d61d7164c231777472d087ea56"}
>
> Couchdb logging shows (with packet sniffer to tell me how big the
> request/responses are:
>
> [info] [<0.2693.2>] 127.0.0.1 - - 'GET'
> /dslcollection/551ffd9dc43cb73c257628d8a31e9e1f/88e2a3606296cc4984b4c6aa9980ad94.jpg?rev=10-d5aa0c311cbfac1b8c7309163625c5d0
> 200
> (This is a 3.5MB GET)
>
> [info] [<0.2692.2>] 127.0.0.1 - - 'PUT'
> /dslcollection-ios/551ffd9dc43cb73c257628d8a31e9e1f?new_edits=false 201
> (3.5MB PUT payload)
>
> Thanks,
>
> Steve
>
>
>
>
>
>
>
>
>
>
> --
> Steven Ringo
>
>
> Jens Alfke wrote:
>>
>> No, an attachment is stored only once in the DB file no matter how many
>> revisions it appears in. The attachment contents are only copied during
>> compaction.
>>
>> --Jens [via iPhone]
>>
>> On Jan 18, 2012, at 6:22 PM, "CGS"<cg...@gmail.com> wrote:
>>
>>> Hi Steven,
>>>
>>> As far as I know, the attachments are still hardcoded in the document and
>>> so, if the document changes its revision, the new document inherits all
>>> the
>>> fields (including its attachments). Therefore, the replication transfers
>>> again the attachment. I suppose this was the way to avoid the loss of the
>>> attachment in case of compaction.
>>>
>>> Relatively recently, there was a discussion to make an attachment to be
>>> held only once per database (or even maybe externally, don't remember
>>> exactly), but I have no knowledge of its outcome. Maybe someone more
>>> knowledgeable will be able to give you more details here.
>>>
>>> For the time being, my suggestion would be you to use external paths
>>> (usually, they are much smaller) and to transfer the files by an external
>>> script.
>>>
>>> CGS
>>>
>>>
>>>
>>>
>
Re: Attachments and replication
Posted by Jens Alfke <je...@couchbase.com>.
On Jan 19, 2012, at 5:42 PM, Steven Ringo wrote:
> I am confused? Do I need to show more about the replication problem I am
> having?
Yes, I think. All you’ve shown is a replication that gets and puts a large document including attachments. But you haven’t shown anything that shows whether or not the attachment should have been copied. A better test would be:
Create document with large attachment on database A
Replicate from A to B
Update document properties on database A without updating the _attachments property
Replicate from A to B
The last step should ideally transfer only the JSON without any attachment bodies.
—Jens
Re: Attachments and replication
Posted by Steven Ringo <go...@stevenringo.com>.
Hi All,
> I agree with Jens that nothing has been posted to show the problem.
I am confused? Do I need to show more about the replication problem I am
having?
Re: Attachments and replication
Posted by Jens Alfke <je...@couchbase.com>.
On Jan 19, 2012, at 10:51 AM, Robert Newson wrote:
> It seems your reply is scrambled for me. it appears you tacked on a
> paragrah to an unmarked quotation from earlier, quite confusing.
Sorry about that. It seems to be a bug in Apple’s Mail app in Lion, where the plain-text body of an HTML reply isn’t adding ASCII quote markers. And the apache.org listserv strips the HTML body, leaving the malformed plain text.
I try to remember to switch messages to plain-text before I post to this list, but too often I forget.
—Jens
Re: Attachments and replication
Posted by Mark Hahn <ma...@hahnca.com>.
> I still can't parse whether there's an outstanding question on this
thread then.
I agree with Jens that nothing has been posted to show the problem.
Re: Attachments and replication
Posted by Robert Newson <rn...@apache.org>.
hm, ok, I still can't parse whether there's an outstanding question on
this thread then.
On 19 January 2012 19:02, Mark Hahn <ma...@hahnca.com> wrote:
> The reply looks good on gmail.
Re: Attachments and replication
Posted by Mark Hahn <ma...@hahnca.com>.
The reply looks good on gmail.
Re: Attachments and replication
Posted by Robert Newson <rn...@apache.org>.
Jens,
It seems your reply is scrambled for me. it appears you tacked on a
paragrah to an unmarked quotation from earlier, quite confusing.
On 19 January 2012 18:44, Jens Alfke <je...@couchbase.com> wrote:
>
> On Jan 18, 2012, at 7:40 PM, Steven Ringo wrote:
>
> In terms of what you describe, I am definitely getting undesired behavior.
>
> It’s possible what I’m describing applies to 1.2 but not 1.1. Most of what I know about the replicator comes from Filipe, and the danger of learning about features from their implementors is that you learn about what they’re working on, not what’s shipped :)
>
> I ran:
> $ curl -H 'Content-Type: application/json' -X POST
> http://localhost:5984/_replicator -d '{"source":
> "http://127.0.0.1:5984/dslcollection", "target":
> "http://127.0.0.1:5984/dslcollection-ios"}'
> {"ok":true,"id":"f36d2bb4d5bdfd3813eb706130001632","rev":"1-c414a5d61d7164c231777472d087ea56”}
>
> So, from what you posted I can’t tell whether the attachment already existed at the target database. (The fact that both dbs are on the same server doesn’t make a difference, of course. The replicator doesn’t care.)
>
> —Jens
Re: Attachments and replication
Posted by Jens Alfke <je...@couchbase.com>.
On Jan 18, 2012, at 7:40 PM, Steven Ringo wrote:
In terms of what you describe, I am definitely getting undesired behavior.
It’s possible what I’m describing applies to 1.2 but not 1.1. Most of what I know about the replicator comes from Filipe, and the danger of learning about features from their implementors is that you learn about what they’re working on, not what’s shipped :)
I ran:
$ curl -H 'Content-Type: application/json' -X POST
http://localhost:5984/_replicator -d '{"source":
"http://127.0.0.1:5984/dslcollection", "target":
"http://127.0.0.1:5984/dslcollection-ios"}'
{"ok":true,"id":"f36d2bb4d5bdfd3813eb706130001632","rev":"1-c414a5d61d7164c231777472d087ea56”}
So, from what you posted I can’t tell whether the attachment already existed at the target database. (The fact that both dbs are on the same server doesn’t make a difference, of course. The replicator doesn’t care.)
—Jens
Re: Attachments and replication
Posted by Steven Ringo <go...@stevenringo.com>.
Jens,
In terms of what you describe, I am definitely getting undesired behaviour.
I am using CouchDB 1.1.1 on Mac OS replicating from the same instance,
as well as IrisCouch (1.1.1 on Ubuntu, I think).
Ultimately need this to work on Mobile couchbase, but experimenting with
replication locally first.
I ran:
$ curl -H 'Content-Type: application/json' -X POST
http://localhost:5984/_replicator -d '{"source":
"http://127.0.0.1:5984/dslcollection", "target":
"http://127.0.0.1:5984/dslcollection-ios"}'
{"ok":true,"id":"f36d2bb4d5bdfd3813eb706130001632","rev":"1-c414a5d61d7164c231777472d087ea56"}
Couchdb logging shows (with packet sniffer to tell me how big the
request/responses are:
[info] [<0.2693.2>] 127.0.0.1 - - 'GET'
/dslcollection/551ffd9dc43cb73c257628d8a31e9e1f/88e2a3606296cc4984b4c6aa9980ad94.jpg?rev=10-d5aa0c311cbfac1b8c7309163625c5d0
200
(This is a 3.5MB GET)
[info] [<0.2692.2>] 127.0.0.1 - - 'PUT'
/dslcollection-ios/551ffd9dc43cb73c257628d8a31e9e1f?new_edits=false 201
(3.5MB PUT payload)
Thanks,
Steve
--
Steven Ringo
Jens Alfke wrote:
> No, an attachment is stored only once in the DB file no matter how many revisions it appears in. The attachment contents are only copied during compaction.
>
> --Jens [via iPhone]
>
> On Jan 18, 2012, at 6:22 PM, "CGS"<cg...@gmail.com> wrote:
>
>> Hi Steven,
>>
>> As far as I know, the attachments are still hardcoded in the document and
>> so, if the document changes its revision, the new document inherits all the
>> fields (including its attachments). Therefore, the replication transfers
>> again the attachment. I suppose this was the way to avoid the loss of the
>> attachment in case of compaction.
>>
>> Relatively recently, there was a discussion to make an attachment to be
>> held only once per database (or even maybe externally, don't remember
>> exactly), but I have no knowledge of its outcome. Maybe someone more
>> knowledgeable will be able to give you more details here.
>>
>> For the time being, my suggestion would be you to use external paths
>> (usually, they are much smaller) and to transfer the files by an external
>> script.
>>
>> CGS
>>
>>
>>
>>
Re: Attachments and replication
Posted by Jens Alfke <je...@couchbase.com>.
No, an attachment is stored only once in the DB file no matter how many revisions it appears in. The attachment contents are only copied during compaction.
--Jens [via iPhone]
On Jan 18, 2012, at 6:22 PM, "CGS" <cg...@gmail.com> wrote:
> Hi Steven,
>
> As far as I know, the attachments are still hardcoded in the document and
> so, if the document changes its revision, the new document inherits all the
> fields (including its attachments). Therefore, the replication transfers
> again the attachment. I suppose this was the way to avoid the loss of the
> attachment in case of compaction.
>
> Relatively recently, there was a discussion to make an attachment to be
> held only once per database (or even maybe externally, don't remember
> exactly), but I have no knowledge of its outcome. Maybe someone more
> knowledgeable will be able to give you more details here.
>
> For the time being, my suggestion would be you to use external paths
> (usually, they are much smaller) and to transfer the files by an external
> script.
>
> CGS
>
>
>
>
> On Thu, Jan 19, 2012 at 2:52 AM, Steven Ringo <go...@stevenringo.com>wrote:
>
>> Hi all,
>>
>> Assuming I have a document with an attachment and some fields, and that
>> this document has been replicated between source and target servers.
>>
>> {
>> "_id": "**551ffd9dc43cb73c257628d8a31e9e**1f",
>> "_rev": "4-**387d5d23b62e398421bddb10e16e74**0f",
>> "foo": "bar",
>> "baz": "quux",
>> "published": false,
>> "_attachments": {
>> "photo.jpg": {
>> "content_type": "image/jpeg",
>> "revpos": 2,
>> "digest": "md5-iOKjYGKWzEmEtMaqmYCtlA=="**,
>> "length": 3622469,
>> "stub": true
>> }
>> }
>> }
>>
>> Let's say I change one of the fields on the source server; say "foo":
>> "bar" to "foo": "bar2". I then do a pull replication.
>>
>> Sniffing the packets, it seems to me the entire document is replicated,
>> including attachments. I was under the impression couchdb would be able to
>> replicate the document independently of the attachments.
>>
>> In other words couchdb would transfer just the "text/json" portion of the
>> document and not the attachment, i.e. less than ~1 kbyte vs. a few
>> megebytes in this case.
>>
>> My experience seems to contradict my assumption. It looks like the digest
>> and revpos fields are used for this, as mentioned here:
>> http://stackoverflow.com/**questions/1825117/what-is-the-**
>> revpos-value-used-for-in-**couchdb-attachments<http://stackoverflow.com/questions/1825117/what-is-the-revpos-value-used-for-in-couchdb-attachments>
>>
>> Is there something I am missing or perhaps a configuration option I am not
>> setting properly?
>>
>> Thanks.
>>
>>
>>
>>
>>
Re: Attachments and replication
Posted by Steven Ringo <go...@stevenringo.com>.
Hi CGS,
Seems strange to me since updates are made to a document by using stubs;
i.e without recreating the attachment. "When you update the document you
must include the attachment stubs or CouchDB will delete the
attachment." - http://wiki.apache.org/couchdb/HTTP_Document_API
The idea of having to use external storage seems is counter to what I
decided to use couchdb for at the outset.
Thanks,
CGS wrote:
> Hi Steven,
>
> As far as I know, the attachments are still hardcoded in the document and
> so, if the document changes its revision, the new document inherits all the
> fields (including its attachments). Therefore, the replication transfers
> again the attachment. I suppose this was the way to avoid the loss of the
> attachment in case of compaction.
>
> Relatively recently, there was a discussion to make an attachment to be
> held only once per database (or even maybe externally, don't remember
> exactly), but I have no knowledge of its outcome. Maybe someone more
> knowledgeable will be able to give you more details here.
>
> For the time being, my suggestion would be you to use external paths
> (usually, they are much smaller) and to transfer the files by an external
> script.
>
> CGS
>
>
>
>
> On Thu, Jan 19, 2012 at 2:52 AM, Steven Ringo<go...@stevenringo.com>wrote:
>
>> Hi all,
>>
>> Assuming I have a document with an attachment and some fields, and that
>> this document has been replicated between source and target servers.
>>
>> {
>> "_id": "**551ffd9dc43cb73c257628d8a31e9e**1f",
>> "_rev": "4-**387d5d23b62e398421bddb10e16e74**0f",
>> "foo": "bar",
>> "baz": "quux",
>> "published": false,
>> "_attachments": {
>> "photo.jpg": {
>> "content_type": "image/jpeg",
>> "revpos": 2,
>> "digest": "md5-iOKjYGKWzEmEtMaqmYCtlA=="**,
>> "length": 3622469,
>> "stub": true
>> }
>> }
>> }
>>
>> Let's say I change one of the fields on the source server; say "foo":
>> "bar" to "foo": "bar2". I then do a pull replication.
>>
>> Sniffing the packets, it seems to me the entire document is replicated,
>> including attachments. I was under the impression couchdb would be able to
>> replicate the document independently of the attachments.
>>
>> In other words couchdb would transfer just the "text/json" portion of the
>> document and not the attachment, i.e. less than ~1 kbyte vs. a few
>> megebytes in this case.
>>
>> My experience seems to contradict my assumption. It looks like the digest
>> and revpos fields are used for this, as mentioned here:
>> http://stackoverflow.com/**questions/1825117/what-is-the-**
>> revpos-value-used-for-in-**couchdb-attachments<http://stackoverflow.com/questions/1825117/what-is-the-revpos-value-used-for-in-couchdb-attachments>
>>
>> Is there something I am missing or perhaps a configuration option I am not
>> setting properly?
>>
>> Thanks.
>>
>>
>>
>>
>>
>
--
Steven Ringo
Re: Attachments and replication
Posted by CGS <cg...@gmail.com>.
Hi Steven,
As far as I know, the attachments are still hardcoded in the document and
so, if the document changes its revision, the new document inherits all the
fields (including its attachments). Therefore, the replication transfers
again the attachment. I suppose this was the way to avoid the loss of the
attachment in case of compaction.
Relatively recently, there was a discussion to make an attachment to be
held only once per database (or even maybe externally, don't remember
exactly), but I have no knowledge of its outcome. Maybe someone more
knowledgeable will be able to give you more details here.
For the time being, my suggestion would be you to use external paths
(usually, they are much smaller) and to transfer the files by an external
script.
CGS
On Thu, Jan 19, 2012 at 2:52 AM, Steven Ringo <go...@stevenringo.com>wrote:
> Hi all,
>
> Assuming I have a document with an attachment and some fields, and that
> this document has been replicated between source and target servers.
>
> {
> "_id": "**551ffd9dc43cb73c257628d8a31e9e**1f",
> "_rev": "4-**387d5d23b62e398421bddb10e16e74**0f",
> "foo": "bar",
> "baz": "quux",
> "published": false,
> "_attachments": {
> "photo.jpg": {
> "content_type": "image/jpeg",
> "revpos": 2,
> "digest": "md5-iOKjYGKWzEmEtMaqmYCtlA=="**,
> "length": 3622469,
> "stub": true
> }
> }
> }
>
> Let's say I change one of the fields on the source server; say "foo":
> "bar" to "foo": "bar2". I then do a pull replication.
>
> Sniffing the packets, it seems to me the entire document is replicated,
> including attachments. I was under the impression couchdb would be able to
> replicate the document independently of the attachments.
>
> In other words couchdb would transfer just the "text/json" portion of the
> document and not the attachment, i.e. less than ~1 kbyte vs. a few
> megebytes in this case.
>
> My experience seems to contradict my assumption. It looks like the digest
> and revpos fields are used for this, as mentioned here:
> http://stackoverflow.com/**questions/1825117/what-is-the-**
> revpos-value-used-for-in-**couchdb-attachments<http://stackoverflow.com/questions/1825117/what-is-the-revpos-value-used-for-in-couchdb-attachments>
>
> Is there something I am missing or perhaps a configuration option I am not
> setting properly?
>
> Thanks.
>
>
>
>
>