You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Nick Wood <nw...@gmail.com> on 2016/03/09 21:29:28 UTC
Multiple database backup strategy
Hello,
I'm looking to back up a CouchDB server with multiple databases. Currently
1,400, but it fluctuates up and down throughout the day as new databases
are added and old ones deleted. ~10% of the databases are written to within
any 5 minute period of time.
Goals
- Maintain a continual off-site snapshot of all databases, preferably no
older than a few seconds (or minutes)
- Be efficient with bandwidth (i.e. not copy the whole database file for
every backup run)
My current solution watches the global _changes feed and fires up a
continuous replication to an off-site server whenever it sees a change. If
it doesn't see a change from a database for 10 minutes, it kills that
replication. This means I only have ~150 active replications running on
average at any given time.
I thought this was a pretty clever approach, but I can't stabilize it.
Replications hang frequently with crashes in the log file. I haven't yet
tracked down the source of the crashes. I'm running the official 1.6.1
docker image as of yesterday so I don't think it would be an erlang issue.
Rather than keep banging my head against these stability issues, I thought
I'd ask to see if anyone else has come up with a clever backup solution
that meets the above goals?
Nick
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
Hi,
If there's a chance that a user can add a single _replicator doc without it being picked up by _db_updates, I think that's a deal breaker. If a user is regularly adding/updating/deleting _replicator docs then, yes, I believe we can say we'll eventually notice.
I did mean 'use couch_event' not 'us ecouch_event', erroneous space bar activity owing to jetlag.
We have the same goal here. Do you agree or disagree that a continuously _active_ replication should simply stay running until descheduled? We obviously agree that an idle (or low activity) replication should be shut down, freeing sockets, during its idle periods.
B.
> On 19 Mar 2016, at 20:31, Adam Kocoloski <ko...@apache.org> wrote:
>
> Hi Bob, comments inline:
>
>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> Hi,
>>
>> The problem is that _db_updates is not guaranteed to see every update, so I think it falls at the first hurdle.
>
> Do you mean to say that a listener of _db_updates is not guaranteed to see every updated *database*? I think it would be helpful for the discussion to describe the scenario in which an updated database permanently fails to show up in the feed. My recollection is that it’s quite byzantine.
>
>> What couch_replicator_manager does in couchdb 2.0 (though not in the version that Cloudant originally contributed) is to us ecouch_event, notice which are to _replicator shards, and trigger management work from that.
>
> Did you mean to say “couch_event”? I assume so. You’re describing how the replicator manager discovers new replication jobs, not how the jobs discover new updates to source databases specified by replication jobs. Seems orthogonal to me unless I missed something.
>
>> Some work I'm embarking on, with a few other devs here at Cloudant, is to enhance the replicator manager to not run all jobs at once and it is indeed the plan to have each of those jobs run for a while, kill them (they checkpoint then close all resources) and reschedule them later. It's TBD whether we'd always strip feed=continuous from those. We _could_ let each job run to completion (i.e, caught up to the source db as of the start of the replication job) but I think we have to be a bit smarter and allow replication jobs that constantly have work to do (i.e, the source db is always busy), to run as they run today, with feed=continuous, unless forcibly ousted by a scheduler due to some configuration concurrency setting.
>
> So I think this is really the crux of the issue. My contention is that permanently occupying a socket for each continuous replication with the same source and mediator is needlessly expensive, and that _db_updates could be an elegant replacement.
>
>> I note for completeness that the work we're planning explicitly includes "multi database" strategies, you'll hopefully be able to make a single _replicator doc that represents your entire intention (e.g, "replicate _all_ dbs from server1 to server2”).
>
> Nice! It’ll be good to hear more about that design as it evolves, particularly in aspects like discovery of newly created source databases and reporting of 403s and other fatal errors.
>
> Adam
Re: Multiple database backup strategy
Posted by Robert Newson <rn...@apache.org>.
Groovy, that's consensus then.
Where we can use _db_updates to know which pending jobs are worth running, we will. If it's a 404, we'll do something less optimal, start the job itself. Using the connection pool as discussed and accounting for and penalising jobs that complete very quickly or perform no useful work by applying a rising retry interval should mitigate some of that cost. Happily, we discussed those kinds of scheduler nuances this week.
> On 20 Mar 2016, at 14:54, Adam Kocoloski <ko...@apache.org> wrote:
>
> I’ll never berate anyone for top-posting (or bottom-posting for that matter). I just follow suit with whatever the current thread is doing — in this, very very clearly top-posting ;)
>
> Thank you for making this distinction clear. Personally I was only ever interested in the first case. Scoping the replicator manager to only learn about _replicator docs on the local cluster through internal APIs is a smart move — I wasn’t suggesting anything different there. I do think we should have an efficient way for a client to learn about the existence of new updates to an arbitrary number of databases on a remote cluster using a single socket.
>
> Adam
>
>> On Mar 20, 2016, at 10:45 AM, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> Final note, we've conflated two uses of /_db_updates that I want to be very clear on;
>>
>> 1) using /_db_updates to detect active source databases of a replication job.
>> 2) using /_db_updates to hear about new/updated/deleted _replicator documents.
>>
>> It was the 2nd case where the unreliability was a concern, since the update frequency is very low, one expects, for _replicator databases.
>>
>>> On 20 Mar 2016, at 14:44, Robert Samuel Newson <rn...@apache.org> wrote:
>>>
>>> (I swear I'll stop soon...)
>>>
>>> Using /_db_updates as a cheap mechanism to detect activity at the source for any database we're interested in is an important optimization. We didn't discuss it this past week as we felt that /_db_updates wasn't sufficiently reliable. We can save a lot of churn in the scheduler by simply not resuming any job unless we have seen an update to the source database.
>>>
>>> B.
>>>
>>>> On 20 Mar 2016, at 14:36, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>
>>>> I missed a point in Adam's earlier post.
>>>>
>>>> The current scheme uses couch_event for runtime changes to _replicator docs but has to read all updates of all _replicator databases at startup. In the steady state it is just receiving couch_event notifications. The /_db_updates option would change that only slightly (we'd read /_db_updates from 0 to find all _replicator databases, rather than reading the changes feed for the node-local 'dbs' database).
>>>>
>>>> CouchDB itself has a single /_replicator database, of course, but the code will consider any database to be a /_replicator database if the name ends that way. i.e, today, if you made a database called foo/_replicator it would be considered a /_replicator database by the system (and we'd inject the ddoc, etc).
>>>>
>>>> B.
>>>>
>>>>> On 20 Mar 2016, at 14:31, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>>
>>>>> Since I'm typing anyway, and haven't yet been dinged for top-posting, I wanted to mention one other optimization we had in mind.
>>>>>
>>>>> Currently each replicator job has its own connection pool. When we introduce the notion that we can stop and restart jobs, those become approximately useless. So we will obvious hoist that 'up' to a higher level and manage connection pools at the manager level.
>>>>>
>>>>> One optimization that seems obvious from the Cloudant perspective is to allow reuse of connections to the same destinations even though they are ostensibly for different domains. That is, a connection to rnewson.cloudant.com is ultimately a connection to lbX.jenever.cloudant.com. This connection could just as easily be used for any other user in the jenever cluster. Thus, if it's idle, we could borrow that connection rather than create a new one.
>>>>>
>>>>>> host rnewson.cloudant.com
>>>>> rnewson.cloudant.com is an alias for jenever.cloudant.com.
>>>>> jenever.cloudant.com is an alias for lb2.jenever.cloudant.com.
>>>>> lb2.jenever.cloudant.com has address 5.153.0.207
>>>>>
>>>>> Rather than add rnewson.cloudant.com > 5.153.0.207 to the pool, we would add lb2.jenever.cloudant.com -> 5.153.0.207 and resolve rnewson.cloudant.com to its ultimate CNAME before consulting the pool.
>>>>>
>>>>> Does this optimization help elsewhere than Cloudant?
>>>>>
>>>>>> On 20 Mar 2016, at 14:22, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>>>
>>>>>> My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
>>>>>>
>>>>>> The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
>>>>>>
>>>>>> One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
>>>>>>
>>>>>> B.
>>>>>>
>>>>>> P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
>>>>>>
>>>>>>> On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>>>>>>>
>>>>>>> When a shard is updated, it'll trigger a "database updated" event. CouchDB
>>>>>>> will hold those updates in memory for a configurable amount of time in
>>>>>>> order to dedupe updates. It'll then cast lists of updated databases to
>>>>>>> nodes which host the relevant _db_updates shards for further deduplication.
>>>>>>> It's only at that point that the updates are persisted. Only a single
>>>>>>> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
>>>>>>> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
>>>>>>> may be a bit tricky for all of them to fail. You'd need coordinated node
>>>>>>> failure. Perhaps something like datacenter power loss. Another possible
>>>>>>> issue is if all the nodes which host a shard range of the _db_updates DB
>>>>>>> are unreachable by the nodes which host a shard range of any other DB. Even
>>>>>>> if it was momentary, it'd cause messages to be dropped from the _db_updates
>>>>>>> feed.
>>>>>>>
>>>>>>> For n=3 DBs, it seems like it'd be difficult for all of those things to go
>>>>>>> wrong (except perhaps in the case of power loss or catastrophic network
>>>>>>> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
>>>>>>> update.
>>>>>>>
>>>>>>>> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Hi Bob, comments inline:
>>>>>>>>
>>>>>>>>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> The problem is that _db_updates is not guaranteed to see every update,
>>>>>>>> so I think it falls at the first hurdle.
>>>>>>>>
>>>>>>>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>>>>>>>> every updated *database*? I think it would be helpful for the discussion to
>>>>>>>> describe the scenario in which an updated database permanently fails to
>>>>>>>> show up in the feed. My recollection is that it’s quite byzantine.
>>>>>>>>
>>>>>>>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>>>>>>>> version that Cloudant originally contributed) is to us ecouch_event, notice
>>>>>>>> which are to _replicator shards, and trigger management work from that.
>>>>>>>>
>>>>>>>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>>>>>>>> replicator manager discovers new replication jobs, not how the jobs
>>>>>>>> discover new updates to source databases specified by replication jobs.
>>>>>>>> Seems orthogonal to me unless I missed something.
>>>>>>>>
>>>>>>>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>>>>>>>> to enhance the replicator manager to not run all jobs at once and it is
>>>>>>>> indeed the plan to have each of those jobs run for a while, kill them (they
>>>>>>>> checkpoint then close all resources) and reschedule them later. It's TBD
>>>>>>>> whether we'd always strip feed=continuous from those. We _could_ let each
>>>>>>>> job run to completion (i.e, caught up to the source db as of the start of
>>>>>>>> the replication job) but I think we have to be a bit smarter and allow
>>>>>>>> replication jobs that constantly have work to do (i.e, the source db is
>>>>>>>> always busy), to run as they run today, with feed=continuous, unless
>>>>>>>> forcibly ousted by a scheduler due to some configuration concurrency
>>>>>>>> setting.
>>>>>>>>
>>>>>>>> So I think this is really the crux of the issue. My contention is that
>>>>>>>> permanently occupying a socket for each continuous replication with the
>>>>>>>> same source and mediator is needlessly expensive, and that _db_updates
>>>>>>>> could be an elegant replacement.
>>>>>>>>
>>>>>>>>> I note for completeness that the work we're planning explicitly
>>>>>>>> includes "multi database" strategies, you'll hopefully be able to make a
>>>>>>>> single _replicator doc that represents your entire intention (e.g,
>>>>>>>> "replicate _all_ dbs from server1 to server2”).
>>>>>>>>
>>>>>>>> Nice! It’ll be good to hear more about that design as it evolves,
>>>>>>>> particularly in aspects like discovery of newly created source databases
>>>>>>>> and reporting of 403s and other fatal errors.
>>>>>>>>
>>>>>>>> Adam
>
Re: Multiple database backup strategy
Posted by Adam Kocoloski <ko...@apache.org>.
I’ll never berate anyone for top-posting (or bottom-posting for that matter). I just follow suit with whatever the current thread is doing — in this, very very clearly top-posting ;)
Thank you for making this distinction clear. Personally I was only ever interested in the first case. Scoping the replicator manager to only learn about _replicator docs on the local cluster through internal APIs is a smart move — I wasn’t suggesting anything different there. I do think we should have an efficient way for a client to learn about the existence of new updates to an arbitrary number of databases on a remote cluster using a single socket.
Adam
> On Mar 20, 2016, at 10:45 AM, Robert Samuel Newson <rn...@apache.org> wrote:
>
> Final note, we've conflated two uses of /_db_updates that I want to be very clear on;
>
> 1) using /_db_updates to detect active source databases of a replication job.
> 2) using /_db_updates to hear about new/updated/deleted _replicator documents.
>
> It was the 2nd case where the unreliability was a concern, since the update frequency is very low, one expects, for _replicator databases.
>
>> On 20 Mar 2016, at 14:44, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> (I swear I'll stop soon...)
>>
>> Using /_db_updates as a cheap mechanism to detect activity at the source for any database we're interested in is an important optimization. We didn't discuss it this past week as we felt that /_db_updates wasn't sufficiently reliable. We can save a lot of churn in the scheduler by simply not resuming any job unless we have seen an update to the source database.
>>
>> B.
>>
>>> On 20 Mar 2016, at 14:36, Robert Samuel Newson <rn...@apache.org> wrote:
>>>
>>> I missed a point in Adam's earlier post.
>>>
>>> The current scheme uses couch_event for runtime changes to _replicator docs but has to read all updates of all _replicator databases at startup. In the steady state it is just receiving couch_event notifications. The /_db_updates option would change that only slightly (we'd read /_db_updates from 0 to find all _replicator databases, rather than reading the changes feed for the node-local 'dbs' database).
>>>
>>> CouchDB itself has a single /_replicator database, of course, but the code will consider any database to be a /_replicator database if the name ends that way. i.e, today, if you made a database called foo/_replicator it would be considered a /_replicator database by the system (and we'd inject the ddoc, etc).
>>>
>>> B.
>>>
>>>> On 20 Mar 2016, at 14:31, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>
>>>> Since I'm typing anyway, and haven't yet been dinged for top-posting, I wanted to mention one other optimization we had in mind.
>>>>
>>>> Currently each replicator job has its own connection pool. When we introduce the notion that we can stop and restart jobs, those become approximately useless. So we will obvious hoist that 'up' to a higher level and manage connection pools at the manager level.
>>>>
>>>> One optimization that seems obvious from the Cloudant perspective is to allow reuse of connections to the same destinations even though they are ostensibly for different domains. That is, a connection to rnewson.cloudant.com is ultimately a connection to lbX.jenever.cloudant.com. This connection could just as easily be used for any other user in the jenever cluster. Thus, if it's idle, we could borrow that connection rather than create a new one.
>>>>
>>>>> host rnewson.cloudant.com
>>>> rnewson.cloudant.com is an alias for jenever.cloudant.com.
>>>> jenever.cloudant.com is an alias for lb2.jenever.cloudant.com.
>>>> lb2.jenever.cloudant.com has address 5.153.0.207
>>>>
>>>> Rather than add rnewson.cloudant.com > 5.153.0.207 to the pool, we would add lb2.jenever.cloudant.com -> 5.153.0.207 and resolve rnewson.cloudant.com to its ultimate CNAME before consulting the pool.
>>>>
>>>> Does this optimization help elsewhere than Cloudant?
>>>>
>>>>> On 20 Mar 2016, at 14:22, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>>
>>>>> My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
>>>>>
>>>>> The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
>>>>>
>>>>> One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
>>>>>
>>>>> B.
>>>>>
>>>>> P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
>>>>>
>>>>> On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>>>>>>
>>>>>> When a shard is updated, it'll trigger a "database updated" event. CouchDB
>>>>>> will hold those updates in memory for a configurable amount of time in
>>>>>> order to dedupe updates. It'll then cast lists of updated databases to
>>>>>> nodes which host the relevant _db_updates shards for further deduplication.
>>>>>> It's only at that point that the updates are persisted. Only a single
>>>>>> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
>>>>>> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
>>>>>> may be a bit tricky for all of them to fail. You'd need coordinated node
>>>>>> failure. Perhaps something like datacenter power loss. Another possible
>>>>>> issue is if all the nodes which host a shard range of the _db_updates DB
>>>>>> are unreachable by the nodes which host a shard range of any other DB. Even
>>>>>> if it was momentary, it'd cause messages to be dropped from the _db_updates
>>>>>> feed.
>>>>>>
>>>>>> For n=3 DBs, it seems like it'd be difficult for all of those things to go
>>>>>> wrong (except perhaps in the case of power loss or catastrophic network
>>>>>> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
>>>>>> update.
>>>>>>
>>>>>> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>>>>
>>>>>>> Hi Bob, comments inline:
>>>>>>>
>>>>>>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> The problem is that _db_updates is not guaranteed to see every update,
>>>>>>> so I think it falls at the first hurdle.
>>>>>>>
>>>>>>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>>>>>>> every updated *database*? I think it would be helpful for the discussion to
>>>>>>> describe the scenario in which an updated database permanently fails to
>>>>>>> show up in the feed. My recollection is that it’s quite byzantine.
>>>>>>>
>>>>>>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>>>>>>> version that Cloudant originally contributed) is to us ecouch_event, notice
>>>>>>> which are to _replicator shards, and trigger management work from that.
>>>>>>>
>>>>>>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>>>>>>> replicator manager discovers new replication jobs, not how the jobs
>>>>>>> discover new updates to source databases specified by replication jobs.
>>>>>>> Seems orthogonal to me unless I missed something.
>>>>>>>
>>>>>>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>>>>>>> to enhance the replicator manager to not run all jobs at once and it is
>>>>>>> indeed the plan to have each of those jobs run for a while, kill them (they
>>>>>>> checkpoint then close all resources) and reschedule them later. It's TBD
>>>>>>> whether we'd always strip feed=continuous from those. We _could_ let each
>>>>>>> job run to completion (i.e, caught up to the source db as of the start of
>>>>>>> the replication job) but I think we have to be a bit smarter and allow
>>>>>>> replication jobs that constantly have work to do (i.e, the source db is
>>>>>>> always busy), to run as they run today, with feed=continuous, unless
>>>>>>> forcibly ousted by a scheduler due to some configuration concurrency
>>>>>>> setting.
>>>>>>>
>>>>>>> So I think this is really the crux of the issue. My contention is that
>>>>>>> permanently occupying a socket for each continuous replication with the
>>>>>>> same source and mediator is needlessly expensive, and that _db_updates
>>>>>>> could be an elegant replacement.
>>>>>>>
>>>>>>>> I note for completeness that the work we're planning explicitly
>>>>>>> includes "multi database" strategies, you'll hopefully be able to make a
>>>>>>> single _replicator doc that represents your entire intention (e.g,
>>>>>>> "replicate _all_ dbs from server1 to server2”).
>>>>>>>
>>>>>>> Nice! It’ll be good to hear more about that design as it evolves,
>>>>>>> particularly in aspects like discovery of newly created source databases
>>>>>>> and reporting of 403s and other fatal errors.
>>>>>>>
>>>>>>> Adam
>>>>>
>>>>
>>>
>>
>
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
Final note, we've conflated two uses of /_db_updates that I want to be very clear on;
1) using /_db_updates to detect active source databases of a replication job.
2) using /_db_updates to hear about new/updated/deleted _replicator documents.
It was the 2nd case where the unreliability was a concern, since the update frequency is very low, one expects, for _replicator databases.
> On 20 Mar 2016, at 14:44, Robert Samuel Newson <rn...@apache.org> wrote:
>
> (I swear I'll stop soon...)
>
> Using /_db_updates as a cheap mechanism to detect activity at the source for any database we're interested in is an important optimization. We didn't discuss it this past week as we felt that /_db_updates wasn't sufficiently reliable. We can save a lot of churn in the scheduler by simply not resuming any job unless we have seen an update to the source database.
>
> B.
>
>> On 20 Mar 2016, at 14:36, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> I missed a point in Adam's earlier post.
>>
>> The current scheme uses couch_event for runtime changes to _replicator docs but has to read all updates of all _replicator databases at startup. In the steady state it is just receiving couch_event notifications. The /_db_updates option would change that only slightly (we'd read /_db_updates from 0 to find all _replicator databases, rather than reading the changes feed for the node-local 'dbs' database).
>>
>> CouchDB itself has a single /_replicator database, of course, but the code will consider any database to be a /_replicator database if the name ends that way. i.e, today, if you made a database called foo/_replicator it would be considered a /_replicator database by the system (and we'd inject the ddoc, etc).
>>
>> B.
>>
>>> On 20 Mar 2016, at 14:31, Robert Samuel Newson <rn...@apache.org> wrote:
>>>
>>> Since I'm typing anyway, and haven't yet been dinged for top-posting, I wanted to mention one other optimization we had in mind.
>>>
>>> Currently each replicator job has its own connection pool. When we introduce the notion that we can stop and restart jobs, those become approximately useless. So we will obvious hoist that 'up' to a higher level and manage connection pools at the manager level.
>>>
>>> One optimization that seems obvious from the Cloudant perspective is to allow reuse of connections to the same destinations even though they are ostensibly for different domains. That is, a connection to rnewson.cloudant.com is ultimately a connection to lbX.jenever.cloudant.com. This connection could just as easily be used for any other user in the jenever cluster. Thus, if it's idle, we could borrow that connection rather than create a new one.
>>>
>>>> host rnewson.cloudant.com
>>> rnewson.cloudant.com is an alias for jenever.cloudant.com.
>>> jenever.cloudant.com is an alias for lb2.jenever.cloudant.com.
>>> lb2.jenever.cloudant.com has address 5.153.0.207
>>>
>>> Rather than add rnewson.cloudant.com > 5.153.0.207 to the pool, we would add lb2.jenever.cloudant.com -> 5.153.0.207 and resolve rnewson.cloudant.com to its ultimate CNAME before consulting the pool.
>>>
>>> Does this optimization help elsewhere than Cloudant?
>>>
>>>> On 20 Mar 2016, at 14:22, Robert Samuel Newson <rn...@apache.org> wrote:
>>>>
>>>> My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
>>>>
>>>> The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
>>>>
>>>> One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
>>>>
>>>> B.
>>>>
>>>> P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
>>>>
>>>> On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>>>>>
>>>>> When a shard is updated, it'll trigger a "database updated" event. CouchDB
>>>>> will hold those updates in memory for a configurable amount of time in
>>>>> order to dedupe updates. It'll then cast lists of updated databases to
>>>>> nodes which host the relevant _db_updates shards for further deduplication.
>>>>> It's only at that point that the updates are persisted. Only a single
>>>>> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
>>>>> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
>>>>> may be a bit tricky for all of them to fail. You'd need coordinated node
>>>>> failure. Perhaps something like datacenter power loss. Another possible
>>>>> issue is if all the nodes which host a shard range of the _db_updates DB
>>>>> are unreachable by the nodes which host a shard range of any other DB. Even
>>>>> if it was momentary, it'd cause messages to be dropped from the _db_updates
>>>>> feed.
>>>>>
>>>>> For n=3 DBs, it seems like it'd be difficult for all of those things to go
>>>>> wrong (except perhaps in the case of power loss or catastrophic network
>>>>> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
>>>>> update.
>>>>>
>>>>> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>>>
>>>>>> Hi Bob, comments inline:
>>>>>>
>>>>>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The problem is that _db_updates is not guaranteed to see every update,
>>>>>> so I think it falls at the first hurdle.
>>>>>>
>>>>>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>>>>>> every updated *database*? I think it would be helpful for the discussion to
>>>>>> describe the scenario in which an updated database permanently fails to
>>>>>> show up in the feed. My recollection is that it’s quite byzantine.
>>>>>>
>>>>>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>>>>>> version that Cloudant originally contributed) is to us ecouch_event, notice
>>>>>> which are to _replicator shards, and trigger management work from that.
>>>>>>
>>>>>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>>>>>> replicator manager discovers new replication jobs, not how the jobs
>>>>>> discover new updates to source databases specified by replication jobs.
>>>>>> Seems orthogonal to me unless I missed something.
>>>>>>
>>>>>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>>>>>> to enhance the replicator manager to not run all jobs at once and it is
>>>>>> indeed the plan to have each of those jobs run for a while, kill them (they
>>>>>> checkpoint then close all resources) and reschedule them later. It's TBD
>>>>>> whether we'd always strip feed=continuous from those. We _could_ let each
>>>>>> job run to completion (i.e, caught up to the source db as of the start of
>>>>>> the replication job) but I think we have to be a bit smarter and allow
>>>>>> replication jobs that constantly have work to do (i.e, the source db is
>>>>>> always busy), to run as they run today, with feed=continuous, unless
>>>>>> forcibly ousted by a scheduler due to some configuration concurrency
>>>>>> setting.
>>>>>>
>>>>>> So I think this is really the crux of the issue. My contention is that
>>>>>> permanently occupying a socket for each continuous replication with the
>>>>>> same source and mediator is needlessly expensive, and that _db_updates
>>>>>> could be an elegant replacement.
>>>>>>
>>>>>>> I note for completeness that the work we're planning explicitly
>>>>>> includes "multi database" strategies, you'll hopefully be able to make a
>>>>>> single _replicator doc that represents your entire intention (e.g,
>>>>>> "replicate _all_ dbs from server1 to server2”).
>>>>>>
>>>>>> Nice! It’ll be good to hear more about that design as it evolves,
>>>>>> particularly in aspects like discovery of newly created source databases
>>>>>> and reporting of 403s and other fatal errors.
>>>>>>
>>>>>> Adam
>>>>
>>>
>>
>
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
(I swear I'll stop soon...)
Using /_db_updates as a cheap mechanism to detect activity at the source for any database we're interested in is an important optimization. We didn't discuss it this past week as we felt that /_db_updates wasn't sufficiently reliable. We can save a lot of churn in the scheduler by simply not resuming any job unless we have seen an update to the source database.
B.
> On 20 Mar 2016, at 14:36, Robert Samuel Newson <rn...@apache.org> wrote:
>
> I missed a point in Adam's earlier post.
>
> The current scheme uses couch_event for runtime changes to _replicator docs but has to read all updates of all _replicator databases at startup. In the steady state it is just receiving couch_event notifications. The /_db_updates option would change that only slightly (we'd read /_db_updates from 0 to find all _replicator databases, rather than reading the changes feed for the node-local 'dbs' database).
>
> CouchDB itself has a single /_replicator database, of course, but the code will consider any database to be a /_replicator database if the name ends that way. i.e, today, if you made a database called foo/_replicator it would be considered a /_replicator database by the system (and we'd inject the ddoc, etc).
>
> B.
>
>> On 20 Mar 2016, at 14:31, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> Since I'm typing anyway, and haven't yet been dinged for top-posting, I wanted to mention one other optimization we had in mind.
>>
>> Currently each replicator job has its own connection pool. When we introduce the notion that we can stop and restart jobs, those become approximately useless. So we will obvious hoist that 'up' to a higher level and manage connection pools at the manager level.
>>
>> One optimization that seems obvious from the Cloudant perspective is to allow reuse of connections to the same destinations even though they are ostensibly for different domains. That is, a connection to rnewson.cloudant.com is ultimately a connection to lbX.jenever.cloudant.com. This connection could just as easily be used for any other user in the jenever cluster. Thus, if it's idle, we could borrow that connection rather than create a new one.
>>
>>> host rnewson.cloudant.com
>> rnewson.cloudant.com is an alias for jenever.cloudant.com.
>> jenever.cloudant.com is an alias for lb2.jenever.cloudant.com.
>> lb2.jenever.cloudant.com has address 5.153.0.207
>>
>> Rather than add rnewson.cloudant.com > 5.153.0.207 to the pool, we would add lb2.jenever.cloudant.com -> 5.153.0.207 and resolve rnewson.cloudant.com to its ultimate CNAME before consulting the pool.
>>
>> Does this optimization help elsewhere than Cloudant?
>>
>>> On 20 Mar 2016, at 14:22, Robert Samuel Newson <rn...@apache.org> wrote:
>>>
>>> My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
>>>
>>> The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
>>>
>>> One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
>>>
>>> B.
>>>
>>> P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
>>>
>>> On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>>>>
>>>> When a shard is updated, it'll trigger a "database updated" event. CouchDB
>>>> will hold those updates in memory for a configurable amount of time in
>>>> order to dedupe updates. It'll then cast lists of updated databases to
>>>> nodes which host the relevant _db_updates shards for further deduplication.
>>>> It's only at that point that the updates are persisted. Only a single
>>>> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
>>>> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
>>>> may be a bit tricky for all of them to fail. You'd need coordinated node
>>>> failure. Perhaps something like datacenter power loss. Another possible
>>>> issue is if all the nodes which host a shard range of the _db_updates DB
>>>> are unreachable by the nodes which host a shard range of any other DB. Even
>>>> if it was momentary, it'd cause messages to be dropped from the _db_updates
>>>> feed.
>>>>
>>>> For n=3 DBs, it seems like it'd be difficult for all of those things to go
>>>> wrong (except perhaps in the case of power loss or catastrophic network
>>>> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
>>>> update.
>>>>
>>>> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>>
>>>>> Hi Bob, comments inline:
>>>>>
>>>>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The problem is that _db_updates is not guaranteed to see every update,
>>>>> so I think it falls at the first hurdle.
>>>>>
>>>>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>>>>> every updated *database*? I think it would be helpful for the discussion to
>>>>> describe the scenario in which an updated database permanently fails to
>>>>> show up in the feed. My recollection is that it’s quite byzantine.
>>>>>
>>>>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>>>>> version that Cloudant originally contributed) is to us ecouch_event, notice
>>>>> which are to _replicator shards, and trigger management work from that.
>>>>>
>>>>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>>>>> replicator manager discovers new replication jobs, not how the jobs
>>>>> discover new updates to source databases specified by replication jobs.
>>>>> Seems orthogonal to me unless I missed something.
>>>>>
>>>>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>>>>> to enhance the replicator manager to not run all jobs at once and it is
>>>>> indeed the plan to have each of those jobs run for a while, kill them (they
>>>>> checkpoint then close all resources) and reschedule them later. It's TBD
>>>>> whether we'd always strip feed=continuous from those. We _could_ let each
>>>>> job run to completion (i.e, caught up to the source db as of the start of
>>>>> the replication job) but I think we have to be a bit smarter and allow
>>>>> replication jobs that constantly have work to do (i.e, the source db is
>>>>> always busy), to run as they run today, with feed=continuous, unless
>>>>> forcibly ousted by a scheduler due to some configuration concurrency
>>>>> setting.
>>>>>
>>>>> So I think this is really the crux of the issue. My contention is that
>>>>> permanently occupying a socket for each continuous replication with the
>>>>> same source and mediator is needlessly expensive, and that _db_updates
>>>>> could be an elegant replacement.
>>>>>
>>>>>> I note for completeness that the work we're planning explicitly
>>>>> includes "multi database" strategies, you'll hopefully be able to make a
>>>>> single _replicator doc that represents your entire intention (e.g,
>>>>> "replicate _all_ dbs from server1 to server2”).
>>>>>
>>>>> Nice! It’ll be good to hear more about that design as it evolves,
>>>>> particularly in aspects like discovery of newly created source databases
>>>>> and reporting of 403s and other fatal errors.
>>>>>
>>>>> Adam
>>>
>>
>
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
I missed a point in Adam's earlier post.
The current scheme uses couch_event for runtime changes to _replicator docs but has to read all updates of all _replicator databases at startup. In the steady state it is just receiving couch_event notifications. The /_db_updates option would change that only slightly (we'd read /_db_updates from 0 to find all _replicator databases, rather than reading the changes feed for the node-local 'dbs' database).
CouchDB itself has a single /_replicator database, of course, but the code will consider any database to be a /_replicator database if the name ends that way. i.e, today, if you made a database called foo/_replicator it would be considered a /_replicator database by the system (and we'd inject the ddoc, etc).
B.
> On 20 Mar 2016, at 14:31, Robert Samuel Newson <rn...@apache.org> wrote:
>
> Since I'm typing anyway, and haven't yet been dinged for top-posting, I wanted to mention one other optimization we had in mind.
>
> Currently each replicator job has its own connection pool. When we introduce the notion that we can stop and restart jobs, those become approximately useless. So we will obvious hoist that 'up' to a higher level and manage connection pools at the manager level.
>
> One optimization that seems obvious from the Cloudant perspective is to allow reuse of connections to the same destinations even though they are ostensibly for different domains. That is, a connection to rnewson.cloudant.com is ultimately a connection to lbX.jenever.cloudant.com. This connection could just as easily be used for any other user in the jenever cluster. Thus, if it's idle, we could borrow that connection rather than create a new one.
>
>> host rnewson.cloudant.com
> rnewson.cloudant.com is an alias for jenever.cloudant.com.
> jenever.cloudant.com is an alias for lb2.jenever.cloudant.com.
> lb2.jenever.cloudant.com has address 5.153.0.207
>
> Rather than add rnewson.cloudant.com > 5.153.0.207 to the pool, we would add lb2.jenever.cloudant.com -> 5.153.0.207 and resolve rnewson.cloudant.com to its ultimate CNAME before consulting the pool.
>
> Does this optimization help elsewhere than Cloudant?
>
>> On 20 Mar 2016, at 14:22, Robert Samuel Newson <rn...@apache.org> wrote:
>>
>> My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
>>
>> The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
>>
>> One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
>>
>> B.
>>
>> P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
>>
>> On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>>>
>>> When a shard is updated, it'll trigger a "database updated" event. CouchDB
>>> will hold those updates in memory for a configurable amount of time in
>>> order to dedupe updates. It'll then cast lists of updated databases to
>>> nodes which host the relevant _db_updates shards for further deduplication.
>>> It's only at that point that the updates are persisted. Only a single
>>> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
>>> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
>>> may be a bit tricky for all of them to fail. You'd need coordinated node
>>> failure. Perhaps something like datacenter power loss. Another possible
>>> issue is if all the nodes which host a shard range of the _db_updates DB
>>> are unreachable by the nodes which host a shard range of any other DB. Even
>>> if it was momentary, it'd cause messages to be dropped from the _db_updates
>>> feed.
>>>
>>> For n=3 DBs, it seems like it'd be difficult for all of those things to go
>>> wrong (except perhaps in the case of power loss or catastrophic network
>>> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
>>> update.
>>>
>>> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>
>>>> Hi Bob, comments inline:
>>>>
>>>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> The problem is that _db_updates is not guaranteed to see every update,
>>>> so I think it falls at the first hurdle.
>>>>
>>>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>>>> every updated *database*? I think it would be helpful for the discussion to
>>>> describe the scenario in which an updated database permanently fails to
>>>> show up in the feed. My recollection is that it’s quite byzantine.
>>>>
>>>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>>>> version that Cloudant originally contributed) is to us ecouch_event, notice
>>>> which are to _replicator shards, and trigger management work from that.
>>>>
>>>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>>>> replicator manager discovers new replication jobs, not how the jobs
>>>> discover new updates to source databases specified by replication jobs.
>>>> Seems orthogonal to me unless I missed something.
>>>>
>>>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>>>> to enhance the replicator manager to not run all jobs at once and it is
>>>> indeed the plan to have each of those jobs run for a while, kill them (they
>>>> checkpoint then close all resources) and reschedule them later. It's TBD
>>>> whether we'd always strip feed=continuous from those. We _could_ let each
>>>> job run to completion (i.e, caught up to the source db as of the start of
>>>> the replication job) but I think we have to be a bit smarter and allow
>>>> replication jobs that constantly have work to do (i.e, the source db is
>>>> always busy), to run as they run today, with feed=continuous, unless
>>>> forcibly ousted by a scheduler due to some configuration concurrency
>>>> setting.
>>>>
>>>> So I think this is really the crux of the issue. My contention is that
>>>> permanently occupying a socket for each continuous replication with the
>>>> same source and mediator is needlessly expensive, and that _db_updates
>>>> could be an elegant replacement.
>>>>
>>>>> I note for completeness that the work we're planning explicitly
>>>> includes "multi database" strategies, you'll hopefully be able to make a
>>>> single _replicator doc that represents your entire intention (e.g,
>>>> "replicate _all_ dbs from server1 to server2”).
>>>>
>>>> Nice! It’ll be good to hear more about that design as it evolves,
>>>> particularly in aspects like discovery of newly created source databases
>>>> and reporting of 403s and other fatal errors.
>>>>
>>>> Adam
>>
>
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
Since I'm typing anyway, and haven't yet been dinged for top-posting, I wanted to mention one other optimization we had in mind.
Currently each replicator job has its own connection pool. When we introduce the notion that we can stop and restart jobs, those become approximately useless. So we will obvious hoist that 'up' to a higher level and manage connection pools at the manager level.
One optimization that seems obvious from the Cloudant perspective is to allow reuse of connections to the same destinations even though they are ostensibly for different domains. That is, a connection to rnewson.cloudant.com is ultimately a connection to lbX.jenever.cloudant.com. This connection could just as easily be used for any other user in the jenever cluster. Thus, if it's idle, we could borrow that connection rather than create a new one.
> host rnewson.cloudant.com
rnewson.cloudant.com is an alias for jenever.cloudant.com.
jenever.cloudant.com is an alias for lb2.jenever.cloudant.com.
lb2.jenever.cloudant.com has address 5.153.0.207
Rather than add rnewson.cloudant.com > 5.153.0.207 to the pool, we would add lb2.jenever.cloudant.com -> 5.153.0.207 and resolve rnewson.cloudant.com to its ultimate CNAME before consulting the pool.
Does this optimization help elsewhere than Cloudant?
> On 20 Mar 2016, at 14:22, Robert Samuel Newson <rn...@apache.org> wrote:
>
> My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
>
> The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
>
> One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
>
> B.
>
> P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
>
> On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>>
>> When a shard is updated, it'll trigger a "database updated" event. CouchDB
>> will hold those updates in memory for a configurable amount of time in
>> order to dedupe updates. It'll then cast lists of updated databases to
>> nodes which host the relevant _db_updates shards for further deduplication.
>> It's only at that point that the updates are persisted. Only a single
>> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
>> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
>> may be a bit tricky for all of them to fail. You'd need coordinated node
>> failure. Perhaps something like datacenter power loss. Another possible
>> issue is if all the nodes which host a shard range of the _db_updates DB
>> are unreachable by the nodes which host a shard range of any other DB. Even
>> if it was momentary, it'd cause messages to be dropped from the _db_updates
>> feed.
>>
>> For n=3 DBs, it seems like it'd be difficult for all of those things to go
>> wrong (except perhaps in the case of power loss or catastrophic network
>> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
>> update.
>>
>> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> Hi Bob, comments inline:
>>>
>>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> The problem is that _db_updates is not guaranteed to see every update,
>>> so I think it falls at the first hurdle.
>>>
>>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>>> every updated *database*? I think it would be helpful for the discussion to
>>> describe the scenario in which an updated database permanently fails to
>>> show up in the feed. My recollection is that it’s quite byzantine.
>>>
>>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>>> version that Cloudant originally contributed) is to us ecouch_event, notice
>>> which are to _replicator shards, and trigger management work from that.
>>>
>>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>>> replicator manager discovers new replication jobs, not how the jobs
>>> discover new updates to source databases specified by replication jobs.
>>> Seems orthogonal to me unless I missed something.
>>>
>>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>>> to enhance the replicator manager to not run all jobs at once and it is
>>> indeed the plan to have each of those jobs run for a while, kill them (they
>>> checkpoint then close all resources) and reschedule them later. It's TBD
>>> whether we'd always strip feed=continuous from those. We _could_ let each
>>> job run to completion (i.e, caught up to the source db as of the start of
>>> the replication job) but I think we have to be a bit smarter and allow
>>> replication jobs that constantly have work to do (i.e, the source db is
>>> always busy), to run as they run today, with feed=continuous, unless
>>> forcibly ousted by a scheduler due to some configuration concurrency
>>> setting.
>>>
>>> So I think this is really the crux of the issue. My contention is that
>>> permanently occupying a socket for each continuous replication with the
>>> same source and mediator is needlessly expensive, and that _db_updates
>>> could be an elegant replacement.
>>>
>>>> I note for completeness that the work we're planning explicitly
>>> includes "multi database" strategies, you'll hopefully be able to make a
>>> single _replicator doc that represents your entire intention (e.g,
>>> "replicate _all_ dbs from server1 to server2”).
>>>
>>> Nice! It’ll be good to hear more about that design as it evolves,
>>> particularly in aspects like discovery of newly created source databases
>>> and reporting of 403s and other fatal errors.
>>>
>>> Adam
>
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
My point is that we can (and currently do) trigger the replication manager on receipt of the database updated event, so it avoids all of the other parts of the sequence you describe which could fail.
The obvious difference, and I suspect this is what motivates Adam's position, is that _db_updates can be called remotely. A solution using /_db_updates as its feed can run somewhere else, it wouldn't even need to be a couchdb cluster. With the current 2.0 scheme, the _replicator db has to live on the nodes performing replication management (and therefore it depends on couch_{btree,file} etc). That's a huge incentive to go the /_db_updates route and it would serve as a model for others like pouchdb that cannot choose to co-locate.
One side-benefit we get from using database updated events from the _replicator shards, though, is that it helps us determine which node will run any particular job. We allocate a job to the lowest live erlang node that hosts the document. If we go with /_db_updates, we'll need some other scheme. That's not a bad thing (indeed, it could be a very good thing), but it would need more thought. While in Seattle we did discuss both directions at some length and believe we'd need some form of leader election system, the leader would then assign (and rebalance) replication jobs across the erlang cluster. I pointed at a proof-of-concept implementation of an algorithm I trust that I wrote a while back at https://github.com/cloudant/sobhuza as a possible starting point.
B.
P.S. I'm using Mail.app and simply replying where it sticks the cursor (at the top), but in other forums I've been berated for top-posting. Should I modify my reply style here?
On 19 Mar 2016, at 21:42, Benjamin Bastian <bb...@apache.org> wrote:
>
> When a shard is updated, it'll trigger a "database updated" event. CouchDB
> will hold those updates in memory for a configurable amount of time in
> order to dedupe updates. It'll then cast lists of updated databases to
> nodes which host the relevant _db_updates shards for further deduplication.
> It's only at that point that the updates are persisted. Only a single
> update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
> n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
> may be a bit tricky for all of them to fail. You'd need coordinated node
> failure. Perhaps something like datacenter power loss. Another possible
> issue is if all the nodes which host a shard range of the _db_updates DB
> are unreachable by the nodes which host a shard range of any other DB. Even
> if it was momentary, it'd cause messages to be dropped from the _db_updates
> feed.
>
> For n=3 DBs, it seems like it'd be difficult for all of those things to go
> wrong (except perhaps in the case of power loss or catastrophic network
> failure). For n=1 DBs, you'd simply need to reboot a node soon after an
> update.
>
> On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
>
>> Hi Bob, comments inline:
>>
>>> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
>> wrote:
>>>
>>> Hi,
>>>
>>> The problem is that _db_updates is not guaranteed to see every update,
>> so I think it falls at the first hurdle.
>>
>> Do you mean to say that a listener of _db_updates is not guaranteed to see
>> every updated *database*? I think it would be helpful for the discussion to
>> describe the scenario in which an updated database permanently fails to
>> show up in the feed. My recollection is that it’s quite byzantine.
>>
>>> What couch_replicator_manager does in couchdb 2.0 (though not in the
>> version that Cloudant originally contributed) is to us ecouch_event, notice
>> which are to _replicator shards, and trigger management work from that.
>>
>> Did you mean to say “couch_event”? I assume so. You’re describing how the
>> replicator manager discovers new replication jobs, not how the jobs
>> discover new updates to source databases specified by replication jobs.
>> Seems orthogonal to me unless I missed something.
>>
>>> Some work I'm embarking on, with a few other devs here at Cloudant, is
>> to enhance the replicator manager to not run all jobs at once and it is
>> indeed the plan to have each of those jobs run for a while, kill them (they
>> checkpoint then close all resources) and reschedule them later. It's TBD
>> whether we'd always strip feed=continuous from those. We _could_ let each
>> job run to completion (i.e, caught up to the source db as of the start of
>> the replication job) but I think we have to be a bit smarter and allow
>> replication jobs that constantly have work to do (i.e, the source db is
>> always busy), to run as they run today, with feed=continuous, unless
>> forcibly ousted by a scheduler due to some configuration concurrency
>> setting.
>>
>> So I think this is really the crux of the issue. My contention is that
>> permanently occupying a socket for each continuous replication with the
>> same source and mediator is needlessly expensive, and that _db_updates
>> could be an elegant replacement.
>>
>>> I note for completeness that the work we're planning explicitly
>> includes "multi database" strategies, you'll hopefully be able to make a
>> single _replicator doc that represents your entire intention (e.g,
>> "replicate _all_ dbs from server1 to server2”).
>>
>> Nice! It’ll be good to hear more about that design as it evolves,
>> particularly in aspects like discovery of newly created source databases
>> and reporting of 403s and other fatal errors.
>>
>> Adam
Re: Multiple database backup strategy
Posted by Benjamin Bastian <bb...@apache.org>.
When a shard is updated, it'll trigger a "database updated" event. CouchDB
will hold those updates in memory for a configurable amount of time in
order to dedupe updates. It'll then cast lists of updated databases to
nodes which host the relevant _db_updates shards for further deduplication.
It's only at that point that the updates are persisted. Only a single
update needs to reach the _db_updates DB. IIRC, _db_updates triggers up to
n^3 (assuming the _db_updates DB and the updated DB have the same N), so it
may be a bit tricky for all of them to fail. You'd need coordinated node
failure. Perhaps something like datacenter power loss. Another possible
issue is if all the nodes which host a shard range of the _db_updates DB
are unreachable by the nodes which host a shard range of any other DB. Even
if it was momentary, it'd cause messages to be dropped from the _db_updates
feed.
For n=3 DBs, it seems like it'd be difficult for all of those things to go
wrong (except perhaps in the case of power loss or catastrophic network
failure). For n=1 DBs, you'd simply need to reboot a node soon after an
update.
On Sat, Mar 19, 2016 at 1:31 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Hi Bob, comments inline:
>
> > On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org>
> wrote:
> >
> > Hi,
> >
> > The problem is that _db_updates is not guaranteed to see every update,
> so I think it falls at the first hurdle.
>
> Do you mean to say that a listener of _db_updates is not guaranteed to see
> every updated *database*? I think it would be helpful for the discussion to
> describe the scenario in which an updated database permanently fails to
> show up in the feed. My recollection is that it’s quite byzantine.
>
> > What couch_replicator_manager does in couchdb 2.0 (though not in the
> version that Cloudant originally contributed) is to us ecouch_event, notice
> which are to _replicator shards, and trigger management work from that.
>
> Did you mean to say “couch_event”? I assume so. You’re describing how the
> replicator manager discovers new replication jobs, not how the jobs
> discover new updates to source databases specified by replication jobs.
> Seems orthogonal to me unless I missed something.
>
> > Some work I'm embarking on, with a few other devs here at Cloudant, is
> to enhance the replicator manager to not run all jobs at once and it is
> indeed the plan to have each of those jobs run for a while, kill them (they
> checkpoint then close all resources) and reschedule them later. It's TBD
> whether we'd always strip feed=continuous from those. We _could_ let each
> job run to completion (i.e, caught up to the source db as of the start of
> the replication job) but I think we have to be a bit smarter and allow
> replication jobs that constantly have work to do (i.e, the source db is
> always busy), to run as they run today, with feed=continuous, unless
> forcibly ousted by a scheduler due to some configuration concurrency
> setting.
>
> So I think this is really the crux of the issue. My contention is that
> permanently occupying a socket for each continuous replication with the
> same source and mediator is needlessly expensive, and that _db_updates
> could be an elegant replacement.
>
> > I note for completeness that the work we're planning explicitly
> includes "multi database" strategies, you'll hopefully be able to make a
> single _replicator doc that represents your entire intention (e.g,
> "replicate _all_ dbs from server1 to server2”).
>
> Nice! It’ll be good to hear more about that design as it evolves,
> particularly in aspects like discovery of newly created source databases
> and reporting of 403s and other fatal errors.
>
> Adam
Re: Multiple database backup strategy
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Bob, comments inline:
> On Mar 19, 2016, at 2:36 PM, Robert Samuel Newson <rn...@apache.org> wrote:
>
> Hi,
>
> The problem is that _db_updates is not guaranteed to see every update, so I think it falls at the first hurdle.
Do you mean to say that a listener of _db_updates is not guaranteed to see every updated *database*? I think it would be helpful for the discussion to describe the scenario in which an updated database permanently fails to show up in the feed. My recollection is that it’s quite byzantine.
> What couch_replicator_manager does in couchdb 2.0 (though not in the version that Cloudant originally contributed) is to us ecouch_event, notice which are to _replicator shards, and trigger management work from that.
Did you mean to say “couch_event”? I assume so. You’re describing how the replicator manager discovers new replication jobs, not how the jobs discover new updates to source databases specified by replication jobs. Seems orthogonal to me unless I missed something.
> Some work I'm embarking on, with a few other devs here at Cloudant, is to enhance the replicator manager to not run all jobs at once and it is indeed the plan to have each of those jobs run for a while, kill them (they checkpoint then close all resources) and reschedule them later. It's TBD whether we'd always strip feed=continuous from those. We _could_ let each job run to completion (i.e, caught up to the source db as of the start of the replication job) but I think we have to be a bit smarter and allow replication jobs that constantly have work to do (i.e, the source db is always busy), to run as they run today, with feed=continuous, unless forcibly ousted by a scheduler due to some configuration concurrency setting.
So I think this is really the crux of the issue. My contention is that permanently occupying a socket for each continuous replication with the same source and mediator is needlessly expensive, and that _db_updates could be an elegant replacement.
> I note for completeness that the work we're planning explicitly includes "multi database" strategies, you'll hopefully be able to make a single _replicator doc that represents your entire intention (e.g, "replicate _all_ dbs from server1 to server2”).
Nice! It’ll be good to hear more about that design as it evolves, particularly in aspects like discovery of newly created source databases and reporting of 403s and other fatal errors.
Adam
Re: Multiple database backup strategy
Posted by Robert Samuel Newson <rn...@apache.org>.
Hi,
The problem is that _db_updates is not guaranteed to see every update, so I think it falls at the first hurdle.
What couch_replicator_manager does in couchdb 2.0 (though not in the version that Cloudant originally contributed) is to us ecouch_event, notice which are to _replicator shards, and trigger management work from that.
Some work I'm embarking on, with a few other devs here at Cloudant, is to enhance the replicator manager to not run all jobs at once and it is indeed the plan to have each of those jobs run for a while, kill them (they checkpoint then close all resources) and reschedule them later. It's TBD whether we'd always strip feed=continuous from those. We _could_ let each job run to completion (i.e, caught up to the source db as of the start of the replication job) but I think we have to be a bit smarter and allow replication jobs that constantly have work to do (i.e, the source db is always busy), to run as they run today, with feed=continuous, unless forcibly ousted by a scheduler due to some configuration concurrency setting.
I note for completeness that the work we're planning explicitly includes "multi database" strategies, you'll hopefully be able to make a single _replicator doc that represents your entire intention (e.g, "replicate _all_ dbs from server1 to server2").
B.
> On 14 Mar 2016, at 02:40, Adam Kocoloski <ko...@apache.org> wrote:
>
>
>> On Mar 10, 2016, at 3:18 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>>>
>>> On 09 Mar 2016, at 21:29, Nick Wood <nw...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I'm looking to back up a CouchDB server with multiple databases. Currently
>>> 1,400, but it fluctuates up and down throughout the day as new databases
>>> are added and old ones deleted. ~10% of the databases are written to within
>>> any 5 minute period of time.
>>>
>>> Goals
>>> - Maintain a continual off-site snapshot of all databases, preferably no
>>> older than a few seconds (or minutes)
>>> - Be efficient with bandwidth (i.e. not copy the whole database file for
>>> every backup run)
>>>
>>> My current solution watches the global _changes feed and fires up a
>>> continuous replication to an off-site server whenever it sees a change. If
>>> it doesn't see a change from a database for 10 minutes, it kills that
>>> replication. This means I only have ~150 active replications running on
>>> average at any given time.
>>
>> How about instead of using continuous replications and killing them,
>> use non-continuous replications based on _db_updates? They end
>> automatically and should use fewer resources then.
>>
>> Best
>> Jan
>> --
>
> In my opinion this is actually a design we should adopt for CouchDB’s own replication manager. Keeping all those _changes listeners running is needlessly expensive now that we have _db_updates.
>
> Adam
Re: Multiple database backup strategy
Posted by Javier Candeira <ja...@candeira.com>.
On 16/03/16 04:55, Adam Kocoloski wrote:
> I’m missing something here Javier. A pull replication mediated on the target server should still be able to remotely hit _db_updates on the source, shouldn’t it?
> I do realize that there’s a potential permissions issue here, when the replication is executed with sufficient privileges to read from the source database but not enough to hit _db_updates.
Maybe I spoke up too soon. Since one of the issues for me is bandwith usage I'd assumed local checking for _db_updates on the source db before starting the remote replication.
Thanks!
J
Re: Multiple database backup strategy
Posted by Adam Kocoloski <ko...@apache.org>.
> On Mar 14, 2016, at 2:11 AM, javier@candeira.com wrote:
>
> On 2016-03-14 13:40, Adam Kocoloski wrote:
>
>>>> My current solution watches the global _changes feed and fires up a
>>>> continuous replication to an off-site server whenever it sees a change. If
>>>> it doesn't see a change from a database for 10 minutes, it kills that
>>>> replication. This means I only have ~150 active replications running on
>>>> average at any given time.
>>> How about instead of using continuous replications and killing them,
>>> use non-continuous replications based on _db_updates? They end
>>> automatically and should use fewer resources then.
>>> Best
>>> Jan
>>> --
>> In my opinion this is actually a design we should adopt for CouchDB’s
>> own replication manager. Keeping all those _changes listeners running
>> is needlessly expensive now that we have _db_updates.
>> Adam
>
> I like Jan's solution, and I'm going to try and use it for a project, but it can't be the only solution, at least for one of my cases. In this case I work with devices behind NAT that need to pull data from updated remote databases with reasonably low latency.
>
> The source database can't start a push replication on _db_updates, because it can't find the target database behind NAT. "Pull" replications have to be started by the target database (the NATted device), which doesn't know whether the source database is updated unless it's on a continuous replication.
I’m missing something here Javier. A pull replication mediated on the target server should still be able to remotely hit _db_updates on the source, shouldn’t it?
I do realize that there’s a potential permissions issue here, when the replication is executed with sufficient privileges to read from the source database but not enough to hit _db_updates.
> Thanks for the high quality, high signal/noise list, btw. I don't usually say anything because I don't think I have anything to add, but I always read with interest.
Agreed! I don’t get to spend as much time on CouchDB as I’d like, but I believe we’ve established a nice balance between Pull Requests, JIRA and dev@ that allows people to participate at whatever level makes the most sense. Cheers,
Adam
Re: Multiple database backup strategy
Posted by ja...@candeira.com.
On 2016-03-14 13:40, Adam Kocoloski wrote:
>>> My current solution watches the global _changes feed and fires up a
>>> continuous replication to an off-site server whenever it sees a
>>> change. If
>>> it doesn't see a change from a database for 10 minutes, it kills that
>>> replication. This means I only have ~150 active replications running
>>> on
>>> average at any given time.
>>
>> How about instead of using continuous replications and killing them,
>> use non-continuous replications based on _db_updates? They end
>> automatically and should use fewer resources then.
>>
>> Best
>> Jan
>> --
>
> In my opinion this is actually a design we should adopt for CouchDB’s
> own replication manager. Keeping all those _changes listeners running
> is needlessly expensive now that we have _db_updates.
>
> Adam
I like Jan's solution, and I'm going to try and use it for a project,
but it can't be the only solution, at least for one of my cases. In this
case I work with devices behind NAT that need to pull data from updated
remote databases with reasonably low latency.
The source database can't start a push replication on _db_updates,
because it can't find the target database behind NAT. "Pull"
replications have to be started by the target database (the NATted
device), which doesn't know whether the source database is updated
unless it's on a continuous replication.
Essentially I'm using CouchDB as a kind of STUN server as well as a
replicated datastore, or via replicated datastore. I can't be the only
one.
So maybe the non-continuous replications based on _db_updates could be
the default for push replications, keeping the current _changes listener
default for pull replications.
My own idiosyncratic glossary for this case:
- source: the updated database, which is written to independently of the
replication
- target: the database that only written to via replication, updated
from the new data in the source
- "pull" updating: the target runs the replication locally, which
connects to the remote source
- "push" updating: the source runs the replication locally, which
connects to the remote target
- "reasonably low latency": if the connection fails for some reason, so
be it, we can work on an offline first basis. But if we can have
subsecond latencies between writing to the source database and
replication to the target, the application feels real-time, and that's
the goal.
Thanks for the high quality, high signal/noise list, btw. I don't
usually say anything because I don't think I have anything to add, but I
always read with interest.
Cheers,
Javier
Re: Multiple database backup strategy
Posted by Adam Kocoloski <ko...@apache.org>.
> On Mar 10, 2016, at 3:18 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On 09 Mar 2016, at 21:29, Nick Wood <nw...@gmail.com> wrote:
>>
>> Hello,
>>
>> I'm looking to back up a CouchDB server with multiple databases. Currently
>> 1,400, but it fluctuates up and down throughout the day as new databases
>> are added and old ones deleted. ~10% of the databases are written to within
>> any 5 minute period of time.
>>
>> Goals
>> - Maintain a continual off-site snapshot of all databases, preferably no
>> older than a few seconds (or minutes)
>> - Be efficient with bandwidth (i.e. not copy the whole database file for
>> every backup run)
>>
>> My current solution watches the global _changes feed and fires up a
>> continuous replication to an off-site server whenever it sees a change. If
>> it doesn't see a change from a database for 10 minutes, it kills that
>> replication. This means I only have ~150 active replications running on
>> average at any given time.
>
> How about instead of using continuous replications and killing them,
> use non-continuous replications based on _db_updates? They end
> automatically and should use fewer resources then.
>
> Best
> Jan
> --
In my opinion this is actually a design we should adopt for CouchDB’s own replication manager. Keeping all those _changes listeners running is needlessly expensive now that we have _db_updates.
Adam
Re: Multiple database backup strategy
Posted by Paul Okstad <po...@gmail.com>.
Nick,
I’m working with many small databases and I haven’t noticed any problems with the default rsync behavior (I use options -avHS for archive mode, verbose, preserve hardlinks, and sparse file efficient). I have a couple of large databases that are also backed up this way and I haven’t noticed any performance problems.
Whatever option you choose, I think it’s really important to choose a simple/primitive non-CouchDB backup solution in case your specialized user of CouchDB has unforeseen side effects. Using a hosted service that is isolated from your current hosting solution is also important for the same reasons. Be conservative when it comes to safeguarding data.
—
Paul Okstad
http://pokstad.com
> On Mar 10, 2016, at 2:23 PM, Nick Wood <nw...@gmail.com> wrote:
>
> Thanks for the suggestions.
>
> @Paul, I did some basic testing with rsync. It seems like the checksuming
> isn't super efficient if I'm trying to maintain up-to-the-minute backups of
> large databases (> 1GB) because the whole file has to be read to checksum
> it. I could just append without checksuming but that doesn't feel safe. Can
> I ask if you're having rsync do checksums/verifying or if you're just
> appending and if you've ever had data corruption issues?
>
> @Jan, giving that a shot. So far so good. The backups aren't quite as
> current as they would be if continuous replications would work, but this
> seems more efficient than rsync. Might have a winner if the crashes stay
> away.
>
> Nick
>
> On Thu, Mar 10, 2016 at 1:18 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>>> On 09 Mar 2016, at 21:29, Nick Wood <nw...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I'm looking to back up a CouchDB server with multiple databases.
>> Currently
>>> 1,400, but it fluctuates up and down throughout the day as new databases
>>> are added and old ones deleted. ~10% of the databases are written to
>> within
>>> any 5 minute period of time.
>>>
>>> Goals
>>> - Maintain a continual off-site snapshot of all databases, preferably no
>>> older than a few seconds (or minutes)
>>> - Be efficient with bandwidth (i.e. not copy the whole database file for
>>> every backup run)
>>>
>>> My current solution watches the global _changes feed and fires up a
>>> continuous replication to an off-site server whenever it sees a change.
>> If
>>> it doesn't see a change from a database for 10 minutes, it kills that
>>> replication. This means I only have ~150 active replications running on
>>> average at any given time.
>>
>> How about instead of using continuous replications and killing them,
>> use non-continuous replications based on _db_updates? They end
>> automatically and should use fewer resources then.
>>
>> Best
>> Jan
>> --
>>
>>
>>
>>>
>>> I thought this was a pretty clever approach, but I can't stabilize it.
>>> Replications hang frequently with crashes in the log file. I haven't yet
>>> tracked down the source of the crashes. I'm running the official 1.6.1
>>> docker image as of yesterday so I don't think it would be an erlang
>> issue.
>>>
>>> Rather than keep banging my head against these stability issues, I
>> thought
>>> I'd ask to see if anyone else has come up with a clever backup solution
>>> that meets the above goals?
>>>
>>> Nick
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>>
Re: Multiple database backup strategy
Posted by Nick Wood <nw...@gmail.com>.
Thanks for the suggestions.
@Paul, I did some basic testing with rsync. It seems like the checksuming
isn't super efficient if I'm trying to maintain up-to-the-minute backups of
large databases (> 1GB) because the whole file has to be read to checksum
it. I could just append without checksuming but that doesn't feel safe. Can
I ask if you're having rsync do checksums/verifying or if you're just
appending and if you've ever had data corruption issues?
@Jan, giving that a shot. So far so good. The backups aren't quite as
current as they would be if continuous replications would work, but this
seems more efficient than rsync. Might have a winner if the crashes stay
away.
Nick
On Thu, Mar 10, 2016 at 1:18 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> > On 09 Mar 2016, at 21:29, Nick Wood <nw...@gmail.com> wrote:
> >
> > Hello,
> >
> > I'm looking to back up a CouchDB server with multiple databases.
> Currently
> > 1,400, but it fluctuates up and down throughout the day as new databases
> > are added and old ones deleted. ~10% of the databases are written to
> within
> > any 5 minute period of time.
> >
> > Goals
> > - Maintain a continual off-site snapshot of all databases, preferably no
> > older than a few seconds (or minutes)
> > - Be efficient with bandwidth (i.e. not copy the whole database file for
> > every backup run)
> >
> > My current solution watches the global _changes feed and fires up a
> > continuous replication to an off-site server whenever it sees a change.
> If
> > it doesn't see a change from a database for 10 minutes, it kills that
> > replication. This means I only have ~150 active replications running on
> > average at any given time.
>
> How about instead of using continuous replications and killing them,
> use non-continuous replications based on _db_updates? They end
> automatically and should use fewer resources then.
>
> Best
> Jan
> --
>
>
>
> >
> > I thought this was a pretty clever approach, but I can't stabilize it.
> > Replications hang frequently with crashes in the log file. I haven't yet
> > tracked down the source of the crashes. I'm running the official 1.6.1
> > docker image as of yesterday so I don't think it would be an erlang
> issue.
> >
> > Rather than keep banging my head against these stability issues, I
> thought
> > I'd ask to see if anyone else has come up with a clever backup solution
> > that meets the above goals?
> >
> > Nick
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
>
Re: Multiple database backup strategy
Posted by Jan Lehnardt <ja...@apache.org>.
> On 09 Mar 2016, at 21:29, Nick Wood <nw...@gmail.com> wrote:
>
> Hello,
>
> I'm looking to back up a CouchDB server with multiple databases. Currently
> 1,400, but it fluctuates up and down throughout the day as new databases
> are added and old ones deleted. ~10% of the databases are written to within
> any 5 minute period of time.
>
> Goals
> - Maintain a continual off-site snapshot of all databases, preferably no
> older than a few seconds (or minutes)
> - Be efficient with bandwidth (i.e. not copy the whole database file for
> every backup run)
>
> My current solution watches the global _changes feed and fires up a
> continuous replication to an off-site server whenever it sees a change. If
> it doesn't see a change from a database for 10 minutes, it kills that
> replication. This means I only have ~150 active replications running on
> average at any given time.
How about instead of using continuous replications and killing them,
use non-continuous replications based on _db_updates? They end
automatically and should use fewer resources then.
Best
Jan
--
>
> I thought this was a pretty clever approach, but I can't stabilize it.
> Replications hang frequently with crashes in the log file. I haven't yet
> tracked down the source of the crashes. I'm running the official 1.6.1
> docker image as of yesterday so I don't think it would be an erlang issue.
>
> Rather than keep banging my head against these stability issues, I thought
> I'd ask to see if anyone else has come up with a clever backup solution
> that meets the above goals?
>
> Nick
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: Multiple database backup strategy
Posted by Paul Okstad <po...@gmail.com>.
Hey Nick,
I run a server with several thousand databases. I found that replicating them continuously to a global DB for backup caused CPU usage to skyrocket. My current solution is to use rsync along with rsync.net to backup my database files via a scheduled cron job. Rsync is really efficient at only copying the differences in files. Rsync and cron, alternatively Rsync and a directory watching script, may meet your needs.
—
Paul Okstad
http://pokstad.com
> On Mar 9, 2016, at 12:29 PM, Nick Wood <nw...@gmail.com> wrote:
>
> Hello,
>
> I'm looking to back up a CouchDB server with multiple databases. Currently
> 1,400, but it fluctuates up and down throughout the day as new databases
> are added and old ones deleted. ~10% of the databases are written to within
> any 5 minute period of time.
>
> Goals
> - Maintain a continual off-site snapshot of all databases, preferably no
> older than a few seconds (or minutes)
> - Be efficient with bandwidth (i.e. not copy the whole database file for
> every backup run)
>
> My current solution watches the global _changes feed and fires up a
> continuous replication to an off-site server whenever it sees a change. If
> it doesn't see a change from a database for 10 minutes, it kills that
> replication. This means I only have ~150 active replications running on
> average at any given time.
>
> I thought this was a pretty clever approach, but I can't stabilize it.
> Replications hang frequently with crashes in the log file. I haven't yet
> tracked down the source of the crashes. I'm running the official 1.6.1
> docker image as of yesterday so I don't think it would be an erlang issue.
>
> Rather than keep banging my head against these stability issues, I thought
> I'd ask to see if anyone else has come up with a clever backup solution
> that meets the above goals?
>
> Nick