You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Daniel Gonzalez <go...@gonvaled.com> on 2014/03/18 12:13:44 UTC
Faster one-time replication doing file-system copy?
Hi,
I have two local couchdbs instances (running on different ports): one
master-instance and one testing-instance. Master is a local-copy of my real
data in the production servers, arriving via replication, but otherwise I
do not use it for anything: it is just a local copy that I am using as
check-point to recreate the local testing databases.
Both have the same data, but I am doing testing on the testing-instance,
so sometimes I want to "reset" the testing-instance to the state of the
master-instance (which, as said, is a copy of the real production data)
BTW, I am using this intermediate master instance because this way I have
always a copy of the production data available; this means that even when I
have no network available, I can reset the testing instance: only the most
recent changes to the production data which could not be replicated because
of the non-existing network connection are missing, and I can live with
that.
I can recreate the testing instance by replicating master -> testing, but
since some of the databases are very big, view recalculation takes hours. I
am trying to implement a faster method by doing:
1. Stop master and testing instances
2. Copy all databases / design documents / views from master to instance
3. Restart master and instance
My question is simply: what are the consequences of recreating databases by
doing a filesystem copy? Is this going to be accepted by couchdb, or is it
mandatory to enter the data via replication / REST?
Thanks and regards,
Daniel Gonzalez
Re: Faster one-time replication doing file-system copy?
Posted by Dave Cottlehuber <dc...@jsonified.com>.
I am pretty sure you can get away without stopping the source, so long as you don’t compact.
Also, if the files are not changing much, you can try using rsync to “catch up”, at the expense of more CPU on the source, but significantly less data to transfer overall.
(ensure partial and in-place are set)
On 18. März 2014 at 13:56:02, Stefan Klein (st.fankl.in@gmail.com) wrote:
> Hi,
>
> from my understanding which might be wrong
>
> 2014-03-18 13:31 GMT+01:00 Daniel Gonzalez :
>
> > Thanks Dave,
> >
> > Yes, I understand that I need to restart it. Actually, I assume that both
> > source and destination must be stopped while copying:
> > - source to avoid data changes while copying, which can lead to
> > inconsistent data being copied
> >
>
> A couchdb datafile is never inconsistent.
> see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
>
>
> > - destination to avoid that two processes (cp and couchdb) are stepping on
> > each writing on the same files.
> >
>
> That's not how filehandles work (at least on Linux)
> open 4 terminals
> terminal 1:
> cd /tmp
> cat > file
> terminal 2:
> tail -f /tmp/file
> terminal 1:
> enter some text
> terminal 3:
> rm /tmp/file
> cat > /tmp/file
> terminal 4:
> tail -f /tmp/file
> terminal 1:
> enter some text
> terminal 3:
> enter some text
>
>
> as long the filehandle is not closed and re-opened it reads and writes to
> the "old" file, although the "old" file doesn't have a directory entry
> anymore.
>
> So it should be fine to just copy over the file and post to _restart after
> the copy finished.
>
> regards,
> Stefan
>
--
Dave Cottlehuber
Sent from my PDP11
Re: Faster one-time replication doing file-system copy?
Posted by Ryan Mohr <ry...@kumu.io>.
For a while now I've wished couchdb had a fork feature to speed up initial
replication times like this.
curl -X POST -d '{"source":"db1","target":"db2"}'
http://localhost:5984/_fork
Basic process would be something like:
1. Do a simple file system copy to create the new db
2. Set the replication checkpoint on the source db
3. [Optional] Trigger a replication to carry over any doc updates that
occurred during the process
I imagine this would be easy to add same host, same couchdb instance...
probably even same host, different instances. Gets a lot trickier when
you need to copy across hosts though.
Re: Faster one-time replication doing file-system copy?
Posted by Mark Hahn <ma...@hahnca.com>.
> ... calculated VIEWS
You can't calculate anything from more than one doc at once. If you could
then the db could become inconsistent.
If I was smart enough I think I could prove that a DB that can't be
inconsistent can be copied at will.
On Tue, Mar 18, 2014 at 9:47 AM, Riyad Kalla <rk...@gmail.com> wrote:
> Mark,
> You copy the live DB file to a secondary server, restart the secondary and
> it updates the views using the difference in new data no problem?
>
> I think Daniel's point is the inconsistency that may exist between core DB
> and calculated VIEWS unless he takes them all together at the same instant
> in time (simulated by stopping the replicated master)
>
>
> On Tue, Mar 18, 2014 at 8:58 AM, Mark Hahn <ma...@hahnca.com> wrote:
>
> > I copy live DBs all the time. Just make sure you don't copy anything
> along
> > with the actual .db file. Let the views rebuild.
> >
> >
> > On Tue, Mar 18, 2014 at 8:10 AM, Daniel Gonzalez <gonvaled@gonvaled.com
> > >wrote:
> >
> > > On Tue, Mar 18, 2014 at 1:48 PM, Stefan Klein <st...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > from my understanding which might be wrong
> > > >
> > > > 2014-03-18 13:31 GMT+01:00 Daniel Gonzalez <go...@gonvaled.com>:
> > > >
> > > > > Thanks Dave,
> > > > >
> > > > > Yes, I understand that I need to restart it. Actually, I assume
> that
> > > both
> > > > > source and destination must be stopped while copying:
> > > > > - source to avoid data changes while copying, which can lead to
> > > > > inconsistent data being copied
> > > > >
> > > >
> > > > A couchdb datafile is never inconsistent.
> > > > see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
> > >
> > >
> > > Mmm. But we are copying several databases / design_docs / view files,
> any
> > > of which can be changing while the long copy operation is running. It
> > just
> > > seems easier to minimize any inconsistency risk to just stop the
> source.
> > > Even if a file is always consistent, how to guarantee that the
> different
> > > files are consistent with each other? I mean:
> > > 1) I copy a database
> > > 2) Just after the copy is finished, a new doc is appended
> > > 3) That doc is processed in one of the views
> > > 4) Now it is the turn to copy the view file
> > >
> > > Suddenly, the destination database has one view with one indexed
> document
> > > which is *not* in the database file. Maybe couchdb is going to get
> > confused
> > > about it when I start it?
> > >
> > > > - destination to avoid that two processes (cp and couchdb) are
> > stepping
> > > > on
> > > > > each writing on the same files.
> > > > >
> > > >
> > > > That's not how filehandles work (at least on Linux)
> > > > open 4 terminals
> > > > terminal 1:
> > > > cd /tmp
> > > > cat > file
> > > > terminal 2:
> > > > tail -f /tmp/file
> > > > terminal 1:
> > > > enter some text
> > > > terminal 3:
> > > > rm /tmp/file
> > > > cat > /tmp/file
> > > > terminal 4:
> > > > tail -f /tmp/file
> > > > terminal 1:
> > > > enter some text
> > > > terminal 3:
> > > > enter some text
> > > >
> > > >
> > > > as long the filehandle is not closed and re-opened it reads and
> writes
> > to
> > > > the "old" file, although the "old" file doesn't have a directory
> entry
> > > > anymore.
> > > >
> > > > So it should be fine to just copy over the file and post to _restart
> > > after
> > > > the copy finished.
> > > >
> > >
> > > Fair enough, but that assumes that couchdb is not reopening files from
> > the
> > > filesystem. How can I be sure of that, without being *intimate*
> familiar
> > > with the couchdb sources? Or is there any guarantee on the
> documentation
> > > that no file reopening is being performed? What about database /
> > > design_docs delete / create operations? It just seems *very* difficult
> to
> > > guarantee that, under any scenario, couchdb is not going to open a file
> > > from the filesystem, one which has just been created by the copy
> > operation,
> > > and which makes the whole system behave badly.
> > >
> > > Stopping the destination instance (and thus not being able to use it,
> not
> > > even by mistake), seems like an easy way to guarantee that no problems
> > are
> > > going to occur. We are talking about a test instance anyway, so
> downtime
> > is
> > > not an issue.
> > >
> > >
> > > > regards,
> > > > Stefan
> > > >
> > >
> > > Thanks!
> > > Daniel
> > >
> >
>
Re: Faster one-time replication doing file-system copy?
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 18. März 2014 at 17:48:01, Riyad Kalla (rkalla@gmail.com) wrote:
> Mark,
> You copy the live DB file to a secondary server, restart the secondary and
> it updates the views using the difference in new data no problem?
Yes; you’ll need to query it of course to trigger that.
> I think Daniel's point is the inconsistency that may exist between core DB
> and calculated VIEWS unless he takes them all together at the same instant
> in time (simulated by stopping the replicated master)
And yes. This is where btrfs or zfs make life too easy for us mere mortals.
--
Dave Cottlehuber
Sent from my PDP11
Re: Faster one-time replication doing file-system copy?
Posted by Riyad Kalla <rk...@gmail.com>.
Mark,
You copy the live DB file to a secondary server, restart the secondary and
it updates the views using the difference in new data no problem?
I think Daniel's point is the inconsistency that may exist between core DB
and calculated VIEWS unless he takes them all together at the same instant
in time (simulated by stopping the replicated master)
On Tue, Mar 18, 2014 at 8:58 AM, Mark Hahn <ma...@hahnca.com> wrote:
> I copy live DBs all the time. Just make sure you don't copy anything along
> with the actual .db file. Let the views rebuild.
>
>
> On Tue, Mar 18, 2014 at 8:10 AM, Daniel Gonzalez <gonvaled@gonvaled.com
> >wrote:
>
> > On Tue, Mar 18, 2014 at 1:48 PM, Stefan Klein <st...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > from my understanding which might be wrong
> > >
> > > 2014-03-18 13:31 GMT+01:00 Daniel Gonzalez <go...@gonvaled.com>:
> > >
> > > > Thanks Dave,
> > > >
> > > > Yes, I understand that I need to restart it. Actually, I assume that
> > both
> > > > source and destination must be stopped while copying:
> > > > - source to avoid data changes while copying, which can lead to
> > > > inconsistent data being copied
> > > >
> > >
> > > A couchdb datafile is never inconsistent.
> > > see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
> >
> >
> > Mmm. But we are copying several databases / design_docs / view files, any
> > of which can be changing while the long copy operation is running. It
> just
> > seems easier to minimize any inconsistency risk to just stop the source.
> > Even if a file is always consistent, how to guarantee that the different
> > files are consistent with each other? I mean:
> > 1) I copy a database
> > 2) Just after the copy is finished, a new doc is appended
> > 3) That doc is processed in one of the views
> > 4) Now it is the turn to copy the view file
> >
> > Suddenly, the destination database has one view with one indexed document
> > which is *not* in the database file. Maybe couchdb is going to get
> confused
> > about it when I start it?
> >
> > > - destination to avoid that two processes (cp and couchdb) are
> stepping
> > > on
> > > > each writing on the same files.
> > > >
> > >
> > > That's not how filehandles work (at least on Linux)
> > > open 4 terminals
> > > terminal 1:
> > > cd /tmp
> > > cat > file
> > > terminal 2:
> > > tail -f /tmp/file
> > > terminal 1:
> > > enter some text
> > > terminal 3:
> > > rm /tmp/file
> > > cat > /tmp/file
> > > terminal 4:
> > > tail -f /tmp/file
> > > terminal 1:
> > > enter some text
> > > terminal 3:
> > > enter some text
> > >
> > >
> > > as long the filehandle is not closed and re-opened it reads and writes
> to
> > > the "old" file, although the "old" file doesn't have a directory entry
> > > anymore.
> > >
> > > So it should be fine to just copy over the file and post to _restart
> > after
> > > the copy finished.
> > >
> >
> > Fair enough, but that assumes that couchdb is not reopening files from
> the
> > filesystem. How can I be sure of that, without being *intimate* familiar
> > with the couchdb sources? Or is there any guarantee on the documentation
> > that no file reopening is being performed? What about database /
> > design_docs delete / create operations? It just seems *very* difficult to
> > guarantee that, under any scenario, couchdb is not going to open a file
> > from the filesystem, one which has just been created by the copy
> operation,
> > and which makes the whole system behave badly.
> >
> > Stopping the destination instance (and thus not being able to use it, not
> > even by mistake), seems like an easy way to guarantee that no problems
> are
> > going to occur. We are talking about a test instance anyway, so downtime
> is
> > not an issue.
> >
> >
> > > regards,
> > > Stefan
> > >
> >
> > Thanks!
> > Daniel
> >
>
Re: Faster one-time replication doing file-system copy?
Posted by Daniel Gonzalez <go...@gonvaled.com>.
On Tue, Mar 18, 2014 at 4:58 PM, Mark Hahn <ma...@hahnca.com> wrote:
> I copy live DBs all the time. Just make sure you don't copy anything along
> with the actual .db file. Let the views rebuild.
>
As you can see in my original email, the *main* point of this exercise is
to avoid view recreation, thus bringing the whole reset time from the hour
range to the minute range.
>From your comment I understand that overwriting a view file on a running
instance is not a good idea, which is what I was fearing.
>
> On Tue, Mar 18, 2014 at 8:10 AM, Daniel Gonzalez <gonvaled@gonvaled.com
> >wrote:
>
> > On Tue, Mar 18, 2014 at 1:48 PM, Stefan Klein <st...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > from my understanding which might be wrong
> > >
> > > 2014-03-18 13:31 GMT+01:00 Daniel Gonzalez <go...@gonvaled.com>:
> > >
> > > > Thanks Dave,
> > > >
> > > > Yes, I understand that I need to restart it. Actually, I assume that
> > both
> > > > source and destination must be stopped while copying:
> > > > - source to avoid data changes while copying, which can lead to
> > > > inconsistent data being copied
> > > >
> > >
> > > A couchdb datafile is never inconsistent.
> > > see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
> >
> >
> > Mmm. But we are copying several databases / design_docs / view files, any
> > of which can be changing while the long copy operation is running. It
> just
> > seems easier to minimize any inconsistency risk to just stop the source.
> > Even if a file is always consistent, how to guarantee that the different
> > files are consistent with each other? I mean:
> > 1) I copy a database
> > 2) Just after the copy is finished, a new doc is appended
> > 3) That doc is processed in one of the views
> > 4) Now it is the turn to copy the view file
> >
> > Suddenly, the destination database has one view with one indexed document
> > which is *not* in the database file. Maybe couchdb is going to get
> confused
> > about it when I start it?
> >
> > > - destination to avoid that two processes (cp and couchdb) are
> stepping
> > > on
> > > > each writing on the same files.
> > > >
> > >
> > > That's not how filehandles work (at least on Linux)
> > > open 4 terminals
> > > terminal 1:
> > > cd /tmp
> > > cat > file
> > > terminal 2:
> > > tail -f /tmp/file
> > > terminal 1:
> > > enter some text
> > > terminal 3:
> > > rm /tmp/file
> > > cat > /tmp/file
> > > terminal 4:
> > > tail -f /tmp/file
> > > terminal 1:
> > > enter some text
> > > terminal 3:
> > > enter some text
> > >
> > >
> > > as long the filehandle is not closed and re-opened it reads and writes
> to
> > > the "old" file, although the "old" file doesn't have a directory entry
> > > anymore.
> > >
> > > So it should be fine to just copy over the file and post to _restart
> > after
> > > the copy finished.
> > >
> >
> > Fair enough, but that assumes that couchdb is not reopening files from
> the
> > filesystem. How can I be sure of that, without being *intimate* familiar
> > with the couchdb sources? Or is there any guarantee on the documentation
> > that no file reopening is being performed? What about database /
> > design_docs delete / create operations? It just seems *very* difficult to
> > guarantee that, under any scenario, couchdb is not going to open a file
> > from the filesystem, one which has just been created by the copy
> operation,
> > and which makes the whole system behave badly.
> >
> > Stopping the destination instance (and thus not being able to use it, not
> > even by mistake), seems like an easy way to guarantee that no problems
> are
> > going to occur. We are talking about a test instance anyway, so downtime
> is
> > not an issue.
> >
> >
> > > regards,
> > > Stefan
> > >
> >
> > Thanks!
> > Daniel
> >
>
Re: Faster one-time replication doing file-system copy?
Posted by Mark Hahn <ma...@hahnca.com>.
I copy live DBs all the time. Just make sure you don't copy anything along
with the actual .db file. Let the views rebuild.
On Tue, Mar 18, 2014 at 8:10 AM, Daniel Gonzalez <go...@gonvaled.com>wrote:
> On Tue, Mar 18, 2014 at 1:48 PM, Stefan Klein <st...@gmail.com>
> wrote:
>
> > Hi,
> >
> > from my understanding which might be wrong
> >
> > 2014-03-18 13:31 GMT+01:00 Daniel Gonzalez <go...@gonvaled.com>:
> >
> > > Thanks Dave,
> > >
> > > Yes, I understand that I need to restart it. Actually, I assume that
> both
> > > source and destination must be stopped while copying:
> > > - source to avoid data changes while copying, which can lead to
> > > inconsistent data being copied
> > >
> >
> > A couchdb datafile is never inconsistent.
> > see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
>
>
> Mmm. But we are copying several databases / design_docs / view files, any
> of which can be changing while the long copy operation is running. It just
> seems easier to minimize any inconsistency risk to just stop the source.
> Even if a file is always consistent, how to guarantee that the different
> files are consistent with each other? I mean:
> 1) I copy a database
> 2) Just after the copy is finished, a new doc is appended
> 3) That doc is processed in one of the views
> 4) Now it is the turn to copy the view file
>
> Suddenly, the destination database has one view with one indexed document
> which is *not* in the database file. Maybe couchdb is going to get confused
> about it when I start it?
>
> > - destination to avoid that two processes (cp and couchdb) are stepping
> > on
> > > each writing on the same files.
> > >
> >
> > That's not how filehandles work (at least on Linux)
> > open 4 terminals
> > terminal 1:
> > cd /tmp
> > cat > file
> > terminal 2:
> > tail -f /tmp/file
> > terminal 1:
> > enter some text
> > terminal 3:
> > rm /tmp/file
> > cat > /tmp/file
> > terminal 4:
> > tail -f /tmp/file
> > terminal 1:
> > enter some text
> > terminal 3:
> > enter some text
> >
> >
> > as long the filehandle is not closed and re-opened it reads and writes to
> > the "old" file, although the "old" file doesn't have a directory entry
> > anymore.
> >
> > So it should be fine to just copy over the file and post to _restart
> after
> > the copy finished.
> >
>
> Fair enough, but that assumes that couchdb is not reopening files from the
> filesystem. How can I be sure of that, without being *intimate* familiar
> with the couchdb sources? Or is there any guarantee on the documentation
> that no file reopening is being performed? What about database /
> design_docs delete / create operations? It just seems *very* difficult to
> guarantee that, under any scenario, couchdb is not going to open a file
> from the filesystem, one which has just been created by the copy operation,
> and which makes the whole system behave badly.
>
> Stopping the destination instance (and thus not being able to use it, not
> even by mistake), seems like an easy way to guarantee that no problems are
> going to occur. We are talking about a test instance anyway, so downtime is
> not an issue.
>
>
> > regards,
> > Stefan
> >
>
> Thanks!
> Daniel
>
Re: Faster one-time replication doing file-system copy?
Posted by Daniel Gonzalez <go...@gonvaled.com>.
On Tue, Mar 18, 2014 at 1:48 PM, Stefan Klein <st...@gmail.com> wrote:
> Hi,
>
> from my understanding which might be wrong
>
> 2014-03-18 13:31 GMT+01:00 Daniel Gonzalez <go...@gonvaled.com>:
>
> > Thanks Dave,
> >
> > Yes, I understand that I need to restart it. Actually, I assume that both
> > source and destination must be stopped while copying:
> > - source to avoid data changes while copying, which can lead to
> > inconsistent data being copied
> >
>
> A couchdb datafile is never inconsistent.
> see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
Mmm. But we are copying several databases / design_docs / view files, any
of which can be changing while the long copy operation is running. It just
seems easier to minimize any inconsistency risk to just stop the source.
Even if a file is always consistent, how to guarantee that the different
files are consistent with each other? I mean:
1) I copy a database
2) Just after the copy is finished, a new doc is appended
3) That doc is processed in one of the views
4) Now it is the turn to copy the view file
Suddenly, the destination database has one view with one indexed document
which is *not* in the database file. Maybe couchdb is going to get confused
about it when I start it?
> - destination to avoid that two processes (cp and couchdb) are stepping
> on
> > each writing on the same files.
> >
>
> That's not how filehandles work (at least on Linux)
> open 4 terminals
> terminal 1:
> cd /tmp
> cat > file
> terminal 2:
> tail -f /tmp/file
> terminal 1:
> enter some text
> terminal 3:
> rm /tmp/file
> cat > /tmp/file
> terminal 4:
> tail -f /tmp/file
> terminal 1:
> enter some text
> terminal 3:
> enter some text
>
>
> as long the filehandle is not closed and re-opened it reads and writes to
> the "old" file, although the "old" file doesn't have a directory entry
> anymore.
>
> So it should be fine to just copy over the file and post to _restart after
> the copy finished.
>
Fair enough, but that assumes that couchdb is not reopening files from the
filesystem. How can I be sure of that, without being *intimate* familiar
with the couchdb sources? Or is there any guarantee on the documentation
that no file reopening is being performed? What about database /
design_docs delete / create operations? It just seems *very* difficult to
guarantee that, under any scenario, couchdb is not going to open a file
from the filesystem, one which has just been created by the copy operation,
and which makes the whole system behave badly.
Stopping the destination instance (and thus not being able to use it, not
even by mistake), seems like an easy way to guarantee that no problems are
going to occur. We are talking about a test instance anyway, so downtime is
not an issue.
> regards,
> Stefan
>
Thanks!
Daniel
Re: Faster one-time replication doing file-system copy?
Posted by Stefan Klein <st...@gmail.com>.
Hi,
from my understanding which might be wrong
2014-03-18 13:31 GMT+01:00 Daniel Gonzalez <go...@gonvaled.com>:
> Thanks Dave,
>
> Yes, I understand that I need to restart it. Actually, I assume that both
> source and destination must be stopped while copying:
> - source to avoid data changes while copying, which can lead to
> inconsistent data being copied
>
A couchdb datafile is never inconsistent.
see https://wiki.apache.org/couchdb/How_to_make_filesystem_backups
> - destination to avoid that two processes (cp and couchdb) are stepping on
> each writing on the same files.
>
That's not how filehandles work (at least on Linux)
open 4 terminals
terminal 1:
cd /tmp
cat > file
terminal 2:
tail -f /tmp/file
terminal 1:
enter some text
terminal 3:
rm /tmp/file
cat > /tmp/file
terminal 4:
tail -f /tmp/file
terminal 1:
enter some text
terminal 3:
enter some text
as long the filehandle is not closed and re-opened it reads and writes to
the "old" file, although the "old" file doesn't have a directory entry
anymore.
So it should be fine to just copy over the file and post to _restart after
the copy finished.
regards,
Stefan
Re: Faster one-time replication doing file-system copy?
Posted by Daniel Gonzalez <go...@gonvaled.com>.
Thanks Dave,
Yes, I understand that I need to restart it. Actually, I assume that both
source and destination must be stopped while copying:
- source to avoid data changes while copying, which can lead to
inconsistent data being copied
- destination to avoid that two processes (cp and couchdb) are stepping on
each writing on the same files.
But that is fine with me.
Using zfs / btrfs is overkill for my use case, because I have no experience
with that and I would anyway need to re-install my system, or move couchdb
databases to another partition, which I do not have. Nice idea to keep in
mind for the future though.
BR,
Daniel
On Tue, Mar 18, 2014 at 12:17 PM, Dave Cottlehuber <dc...@jsonified.com>wrote:
> Hey Daniel,
>
> copying is fine. But you *will* need to restart couchdb if you pull
> the rug out from under its feet, aka change the file it may have
> cached data from.
>
> If you can I strongly recommend using something like zfs or btrfs to
> make switching back to snapshots super easy. It's a blast!
>
> A+
> Dave
>
> On 18 March 2014 12:13, Daniel Gonzalez <go...@gonvaled.com> wrote:
> > Hi,
> >
> > I have two local couchdbs instances (running on different ports): one
> > master-instance and one testing-instance. Master is a local-copy of my
> real
> > data in the production servers, arriving via replication, but otherwise I
> > do not use it for anything: it is just a local copy that I am using as
> > check-point to recreate the local testing databases.
> >
> > Both have the same data, but I am doing testing on the testing-instance,
> > so sometimes I want to "reset" the testing-instance to the state of the
> > master-instance (which, as said, is a copy of the real production data)
> >
> > BTW, I am using this intermediate master instance because this way I have
> > always a copy of the production data available; this means that even
> when I
> > have no network available, I can reset the testing instance: only the
> most
> > recent changes to the production data which could not be replicated
> because
> > of the non-existing network connection are missing, and I can live with
> > that.
> >
> > I can recreate the testing instance by replicating master -> testing, but
> > since some of the databases are very big, view recalculation takes
> hours. I
> > am trying to implement a faster method by doing:
> >
> > 1. Stop master and testing instances
> > 2. Copy all databases / design documents / views from master to instance
> > 3. Restart master and instance
> >
> > My question is simply: what are the consequences of recreating databases
> by
> > doing a filesystem copy? Is this going to be accepted by couchdb, or is
> it
> > mandatory to enter the data via replication / REST?
> >
> > Thanks and regards,
> > Daniel Gonzalez
>
Re: Faster one-time replication doing file-system copy?
Posted by Dave Cottlehuber <dc...@jsonified.com>.
Hey Daniel,
copying is fine. But you *will* need to restart couchdb if you pull
the rug out from under its feet, aka change the file it may have
cached data from.
If you can I strongly recommend using something like zfs or btrfs to
make switching back to snapshots super easy. It's a blast!
A+
Dave
On 18 March 2014 12:13, Daniel Gonzalez <go...@gonvaled.com> wrote:
> Hi,
>
> I have two local couchdbs instances (running on different ports): one
> master-instance and one testing-instance. Master is a local-copy of my real
> data in the production servers, arriving via replication, but otherwise I
> do not use it for anything: it is just a local copy that I am using as
> check-point to recreate the local testing databases.
>
> Both have the same data, but I am doing testing on the testing-instance,
> so sometimes I want to "reset" the testing-instance to the state of the
> master-instance (which, as said, is a copy of the real production data)
>
> BTW, I am using this intermediate master instance because this way I have
> always a copy of the production data available; this means that even when I
> have no network available, I can reset the testing instance: only the most
> recent changes to the production data which could not be replicated because
> of the non-existing network connection are missing, and I can live with
> that.
>
> I can recreate the testing instance by replicating master -> testing, but
> since some of the databases are very big, view recalculation takes hours. I
> am trying to implement a faster method by doing:
>
> 1. Stop master and testing instances
> 2. Copy all databases / design documents / views from master to instance
> 3. Restart master and instance
>
> My question is simply: what are the consequences of recreating databases by
> doing a filesystem copy? Is this going to be accepted by couchdb, or is it
> mandatory to enter the data via replication / REST?
>
> Thanks and regards,
> Daniel Gonzalez