You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Sam Wen <Sa...@uts.edu.au> on 2012/03/22 04:25:34 UTC
Replication but missing Documents
Hi all,
Our production version is 1.1.0 with "417876 documents" on it and we built 2 brand new servers with 1.1.1
There are 2 ways to migrate the data,
1) Just copy the db files from the OS to another one. The "number of documents" on the new server will match the old one.
2) Replication between old couchdb and new couchdb, The "number of documents" on the new server always hundreds less then the old one. No matter pull or push replication.
Replication between 2 new servers A and B ( A server with copied db files) also can't match the "number of documents" with each other.
Do anyone has similar issue or how can I investigate? There is no any error report for the replication.
Best Regards
Sam Wen
System Administrator
Faculty of Engineering and IT
University of Technology, Sydney
PH: 9514 2374
Em: sam.wen@uts.edu.au
UTS CRICOS Provider Code: 00099F
DISCLAIMER: This email message and any accompanying attachments may contain confidential information.
If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or
attachments. If you have received this message in error, please notify the sender immediately and delete
this message. Any views expressed in this message are those of the individual sender, except where the
sender expressly, and with authority, states them to be the views of the University of Technology Sydney.
Before opening any attachments, please check them for viruses and defects.
Think. Green. Do.
Please consider the environment before printing this email.
Re: Replication but missing Documents
Posted by Matthew Woodward <ma...@mattwoodward.com>.
The other thing I've run into is permissions. Make sure when you do the
replication that you have the correct user/pass indicated in the source and
destination URL as needed. Particularly where design documents are
concerned I've seen cases where everything *but* the design documents
copied over until I had the credentials in the URLs.
On Wed, Mar 21, 2012 at 10:12 PM, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, Sam. Two thoughts:
>
> 1. Confirm whether you have a .validate_doc_update function in any
> design documents (_design/*) anywhere in the databases.
> 2. Query both data sets: _all_docs and save the output to two files.
> Diff the two files and investigate the output (or share it to the
> community). Do the missing documents give any clue about why that
> might be so?
>
> Good luck!
>
> On Thu, Mar 22, 2012 at 3:25 AM, Sam Wen <Sa...@uts.edu.au> wrote:
> > Hi all,
> >
> > Our production version is 1.1.0 with "417876 documents" on it and we
> built 2 brand new servers with 1.1.1
> >
> > There are 2 ways to migrate the data,
> > 1) Just copy the db files from the OS to another one. The "number of
> documents" on the new server will match the old one.
> > 2) Replication between old couchdb and new couchdb, The "number of
> documents" on the new server always hundreds less then the old one. No
> matter pull or push replication.
> >
> > Replication between 2 new servers A and B ( A server with copied db
> files) also can't match the "number of documents" with each other.
> >
> > Do anyone has similar issue or how can I investigate? There is no any
> error report for the replication.
> >
> > Best Regards
> >
> > Sam Wen
> > System Administrator
> > Faculty of Engineering and IT
> > University of Technology, Sydney
> > PH: 9514 2374
> > Em: sam.wen@uts.edu.au
> >
> > UTS CRICOS Provider Code: 00099F
> > DISCLAIMER: This email message and any accompanying attachments may
> contain confidential information.
> > If you are not the intended recipient, do not read, use, disseminate,
> distribute or copy this message or
> > attachments. If you have received this message in error, please notify
> the sender immediately and delete
> > this message. Any views expressed in this message are those of the
> individual sender, except where the
> > sender expressly, and with authority, states them to be the views of the
> University of Technology Sydney.
> > Before opening any attachments, please check them for viruses and
> defects.
> >
> > Think. Green. Do.
> >
> > Please consider the environment before printing this email.
>
>
>
> --
> Iris Couch
>
--
Matthew Woodward
matt@mattwoodward.com
http://blog.mattwoodward.com
identi.ca / Twitter: @mpwoodward
Please do not send me proprietary file formats such as Word, PowerPoint,
etc. as attachments.
http://www.gnu.org/philosophy/no-word-attachments.html
RE: Replication but missing Documents
Posted by Sam Wen <Sa...@uts.edu.au>.
Hi Jason,
Thank you! That's exactly the way I can find out the missing documents with querying all_docs.
Try to investigate why now.
Regards
Sam
-----Original Message-----
From: Jason Smith [mailto:jhs@iriscouch.com]
Sent: Thursday, 22 March 2012 4:12 PM
To: user@couchdb.apache.org
Subject: Re: Replication but missing Documents
Hi, Sam. Two thoughts:
1. Confirm whether you have a .validate_doc_update function in any
design documents (_design/*) anywhere in the databases.
2. Query both data sets: _all_docs and save the output to two files.
Diff the two files and investigate the output (or share it to the
community). Do the missing documents give any clue about why that
might be so?
Good luck!
On Thu, Mar 22, 2012 at 3:25 AM, Sam Wen <Sa...@uts.edu.au> wrote:
> Hi all,
>
> Our production version is 1.1.0 with "417876 documents" on it and we built 2 brand new servers with 1.1.1
>
> There are 2 ways to migrate the data,
> 1) Just copy the db files from the OS to another one. The "number of documents" on the new server will match the old one.
> 2) Replication between old couchdb and new couchdb, The "number of documents" on the new server always hundreds less then the old one. No matter pull or push replication.
>
> Replication between 2 new servers A and B ( A server with copied db files) also can't match the "number of documents" with each other.
>
> Do anyone has similar issue or how can I investigate? There is no any error report for the replication.
>
> Best Regards
>
> Sam Wen
> System Administrator
> Faculty of Engineering and IT
> University of Technology, Sydney
> PH: 9514 2374
> Em: sam.wen@uts.edu.au
>
> UTS CRICOS Provider Code: 00099F
> DISCLAIMER: This email message and any accompanying attachments may contain confidential information.
> If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or
> attachments. If you have received this message in error, please notify the sender immediately and delete
> this message. Any views expressed in this message are those of the individual sender, except where the
> sender expressly, and with authority, states them to be the views of the University of Technology Sydney.
> Before opening any attachments, please check them for viruses and defects.
>
> Think. Green. Do.
>
> Please consider the environment before printing this email.
--
Iris Couch
UTS CRICOS Provider Code: 00099F
DISCLAIMER: This email message and any accompanying attachments may contain confidential information.
If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or
attachments. If you have received this message in error, please notify the sender immediately and delete
this message. Any views expressed in this message are those of the individual sender, except where the
sender expressly, and with authority, states them to be the views of the University of Technology Sydney.
Before opening any attachments, please check them for viruses and defects.
Think. Green. Do.
Please consider the environment before printing this email.
Re: Replication but missing Documents
Posted by Jason Smith <jh...@iriscouch.com>.
Hi, Sam. Two thoughts:
1. Confirm whether you have a .validate_doc_update function in any
design documents (_design/*) anywhere in the databases.
2. Query both data sets: _all_docs and save the output to two files.
Diff the two files and investigate the output (or share it to the
community). Do the missing documents give any clue about why that
might be so?
Good luck!
On Thu, Mar 22, 2012 at 3:25 AM, Sam Wen <Sa...@uts.edu.au> wrote:
> Hi all,
>
> Our production version is 1.1.0 with "417876 documents" on it and we built 2 brand new servers with 1.1.1
>
> There are 2 ways to migrate the data,
> 1) Just copy the db files from the OS to another one. The "number of documents" on the new server will match the old one.
> 2) Replication between old couchdb and new couchdb, The "number of documents" on the new server always hundreds less then the old one. No matter pull or push replication.
>
> Replication between 2 new servers A and B ( A server with copied db files) also can't match the "number of documents" with each other.
>
> Do anyone has similar issue or how can I investigate? There is no any error report for the replication.
>
> Best Regards
>
> Sam Wen
> System Administrator
> Faculty of Engineering and IT
> University of Technology, Sydney
> PH: 9514 2374
> Em: sam.wen@uts.edu.au
>
> UTS CRICOS Provider Code: 00099F
> DISCLAIMER: This email message and any accompanying attachments may contain confidential information.
> If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or
> attachments. If you have received this message in error, please notify the sender immediately and delete
> this message. Any views expressed in this message are those of the individual sender, except where the
> sender expressly, and with authority, states them to be the views of the University of Technology Sydney.
> Before opening any attachments, please check them for viruses and defects.
>
> Think. Green. Do.
>
> Please consider the environment before printing this email.
--
Iris Couch
Re: Replication but missing Documents
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 22 March 2012 04:25, Sam Wen <Sa...@uts.edu.au> wrote:
> Hi all,
>
> Our production version is 1.1.0 with "417876 documents" on it and we built 2 brand new servers with 1.1.1
>
> There are 2 ways to migrate the data,
> 1) Just copy the db files from the OS to another one. The "number of documents" on the new server will match the old one.
> 2) Replication between old couchdb and new couchdb, The "number of documents" on the new server always hundreds less then the old one. No matter pull or push replication.
>
> Replication between 2 new servers A and B ( A server with copied db files) also can't match the "number of documents" with each other.
>
> Do anyone has similar issue or how can I investigate? There is no any error report for the replication.
>
> Best Regards
>
> Sam Wen
> System Administrator
> Faculty of Engineering and IT
> University of Technology, Sydney
> PH: 9514 2374
> Em: sam.wen@uts.edu.au
>
> UTS CRICOS Provider Code: 00099F
> DISCLAIMER: This email message and any accompanying attachments may contain confidential information.
> If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or
> attachments. If you have received this message in error, please notify the sender immediately and delete
> this message. Any views expressed in this message are those of the individual sender, except where the
> sender expressly, and with authority, states them to be the views of the University of Technology Sydney.
> Before opening any attachments, please check them for viruses and defects.
>
> Think. Green. Do.
>
> Please consider the environment before printing this email.
Personally, I'd rsync the .couch files across and then compact seeing
as it's all local.
If you replicate, make sure you are passing admin credentials into the
target so that ddocs can be written.
In both cases, don't forget _users and any local.ini changes you
might have made.
Other than that I'm awaiting the news of your diff for next guesses.
A+
Dave