You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Bharat Bharat <Bh...@citrix.com> on 2010/04/20 21:08:09 UTC
Couchdb 0.11 replication issue
Hi, I am seeing a very bizzare issue with couchdb 0.11 replication.
When I try to replicate a database from my local machine (mac OSX) to a
remote machine( on a different network), it replicates all documents but
size of the document on the remote machine goes up around 10 times and it
eventually fails.
eg: Host machine: 450 docs, ~45 MB
Target machine after replication: 450 docs, ~ 400 MB
Interestingly, the other way works fine. Ie, if I try to pull this database
from host machine to this remote machine, it replicates fine.
Has anyone encountered this or something similar? I dont have big
attachments.
Thanks very much!
Re: Couchdb 0.11 replication issue
Posted by Adam Kocoloski <ko...@apache.org>.
Actually, it's hard to make an a priori calculation about the size of the target DB. It's true that replication is like compaction in that it removes old MVCC revisions. On the other hand, file size on disk is also driven by the size of write batches. If the target is fast relative to the source you many end in a situation where the the replication is writing one document at a time on the target, which is a worst-case scenario for file sizes.
Compaction uses constant-size batches, so it's not subject to this variability. Compacting the target DB will restore the smaller file size. Best,
Adam
On Apr 22, 2010, at 5:35 AM, Filipe David Manana wrote:
> Paul,
>
> Usually it's the other way around. The replica DB usually stays smaller than
> the source DB (unless the target DB was already populated). This is so
> because doing a replica is similar to a DB compaction in many ways.
>
> On Thu, Apr 22, 2010 at 6:26 AM, Paul Bonser <pi...@paulbonser.com> wrote:
>
>> I think that's pretty typical behavior, since it's append-only
>> re-building any views you have. Try compacting the views and database
>> and see if it's still bigger. My guess is that it won't be.
>>
>> I have a DB which did about the same thing. I replicated from one
>> machine to another, and it grew from 209MB to a little less than a GB,
>> then after compacting, it's back down to 209MB. Though in my case the
>> same thing happened during my initial insert of the data, too, because
>> I was reading from the DB as I was doing the inserts (and thus the
>> views were being rebuilt, ballooning the file).
>>
>> On Tue, Apr 20, 2010 at 2:08 PM, Bharat Bharat <Bh...@citrix.com>
>> wrote:
>>>
>>> Hi, I am seeing a very bizzare issue with couchdb 0.11 replication.
>>>
>>> When I try to replicate a database from my local machine (mac OSX) to a
>>> remote machine( on a different network), it replicates all documents but
>>> size of the document on the remote machine goes up around 10 times and it
>>> eventually fails.
>>>
>>> eg: Host machine: 450 docs, ~45 MB
>>> Target machine after replication: 450 docs, ~ 400 MB
>>>
>>> Interestingly, the other way works fine. Ie, if I try to pull this
>> database
>>> from host machine to this remote machine, it replicates fine.
>>>
>>> Has anyone encountered this or something similar? I dont have big
>>> attachments.
>>>
>>> Thanks very much!
>>
>>
>>
>> --
>> Paul Bonser
>> http://probablyprogramming.com
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
Re: Couchdb 0.11 replication issue
Posted by Filipe David Manana <fd...@gmail.com>.
Paul,
Usually it's the other way around. The replica DB usually stays smaller than
the source DB (unless the target DB was already populated). This is so
because doing a replica is similar to a DB compaction in many ways.
On Thu, Apr 22, 2010 at 6:26 AM, Paul Bonser <pi...@paulbonser.com> wrote:
> I think that's pretty typical behavior, since it's append-only
> re-building any views you have. Try compacting the views and database
> and see if it's still bigger. My guess is that it won't be.
>
> I have a DB which did about the same thing. I replicated from one
> machine to another, and it grew from 209MB to a little less than a GB,
> then after compacting, it's back down to 209MB. Though in my case the
> same thing happened during my initial insert of the data, too, because
> I was reading from the DB as I was doing the inserts (and thus the
> views were being rebuilt, ballooning the file).
>
> On Tue, Apr 20, 2010 at 2:08 PM, Bharat Bharat <Bh...@citrix.com>
> wrote:
> >
> > Hi, I am seeing a very bizzare issue with couchdb 0.11 replication.
> >
> > When I try to replicate a database from my local machine (mac OSX) to a
> > remote machine( on a different network), it replicates all documents but
> > size of the document on the remote machine goes up around 10 times and it
> > eventually fails.
> >
> > eg: Host machine: 450 docs, ~45 MB
> > Target machine after replication: 450 docs, ~ 400 MB
> >
> > Interestingly, the other way works fine. Ie, if I try to pull this
> database
> > from host machine to this remote machine, it replicates fine.
> >
> > Has anyone encountered this or something similar? I dont have big
> > attachments.
> >
> > Thanks very much!
>
>
>
> --
> Paul Bonser
> http://probablyprogramming.com
>
--
Filipe David Manana,
fdmanana@gmail.com
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: Couchdb 0.11 replication issue
Posted by Paul Bonser <pi...@paulbonser.com>.
I think that's pretty typical behavior, since it's append-only
re-building any views you have. Try compacting the views and database
and see if it's still bigger. My guess is that it won't be.
I have a DB which did about the same thing. I replicated from one
machine to another, and it grew from 209MB to a little less than a GB,
then after compacting, it's back down to 209MB. Though in my case the
same thing happened during my initial insert of the data, too, because
I was reading from the DB as I was doing the inserts (and thus the
views were being rebuilt, ballooning the file).
On Tue, Apr 20, 2010 at 2:08 PM, Bharat Bharat <Bh...@citrix.com> wrote:
>
> Hi, I am seeing a very bizzare issue with couchdb 0.11 replication.
>
> When I try to replicate a database from my local machine (mac OSX) to a
> remote machine( on a different network), it replicates all documents but
> size of the document on the remote machine goes up around 10 times and it
> eventually fails.
>
> eg: Host machine: 450 docs, ~45 MB
> Target machine after replication: 450 docs, ~ 400 MB
>
> Interestingly, the other way works fine. Ie, if I try to pull this database
> from host machine to this remote machine, it replicates fine.
>
> Has anyone encountered this or something similar? I dont have big
> attachments.
>
> Thanks very much!
--
Paul Bonser
http://probablyprogramming.com
Re: Included docs in 0.11
Posted by Jochen Krause <jk...@googlemail.com>.
Hi, I am a colleague of Moritz. The typo is only in the mail, but it was still my user error. To make lists work with include_docs you have to access the "doc" object of the row, not the "value" object as in other cases. This is obvious if you look at the result of a view with include_docs. I had expected a magic replace of the value with the object and did not look at the view output in detail ...
Here is some example code to possibly help others avoid that same mistake:
Accessing a doc in a list when the doc is emitted by the view:
provides("xml", function() {
// loop over all rows
while (row = getRow()) {
var solution = <solution/>;
solution.title = row.value.name; //// access with = row.value
solution.id = row.key;
Accessing a doc in a list when the doc is included into the view with include_docs=true:
provides("xml", function() {
// loop over all rows
while (row = getRow()) {
var solution = <solution/>;
solution.title = row.doc.name; //// access with = row.doc
solution.id = row.key;
Cheers
--jo
On :Apr 21, 2010 8:45:08 am J Chris Anderson wrote:
I would like to use the result of the view in a list, e.g. get =
/db/_design/mydesign/_list/mylist/myview?key=3D"foo"&included_docs=3Dtrue
I want to be sure this isn't a basic typo. I think you mean =
include_docs=3Dtrue
(not included...) I'm pretty sure this works.=
Re: Couchdb 0.11 replication issue
Posted by Filipe David Manana <fd...@gmail.com>.
Hi,
Do you have large attachments (bigger than 1Mb in your source DB)? If so,
then it's normal. There's a ticket/patch (
https://issues.apache.org/jira/browse/COUCHDB-639) that fixes the push
replication when docs have large attachments.
However that patch was applied on trunk only, therefore not on the 0.11
release.
Can you test with trunk?
cheers
On Tue, Apr 20, 2010 at 8:08 PM, Bharat Bharat <Bh...@citrix.com>wrote:
> Hi, I am seeing a very bizzare issue with couchdb 0.11 replication.
>
> When I try to replicate a database from my local machine (mac OSX) to a
> remote machine( on a different network), it replicates all documents but
> size of the document on the remote machine goes up around 10 times and it
> eventually fails.
>
> eg: Host machine: 450 docs, ~45 MB
> Target machine after replication: 450 docs, ~ 400 MB
>
> Interestingly, the other way works fine. Ie, if I try to pull this database
> from host machine to this remote machine, it replicates fine.
>
> Has anyone encountered this or something similar? I dont have big
> attachments.
>
> Thanks very much!
>
--
Filipe David Manana,
fdmanana@gmail.com
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."