Best Choice for redundant production environment?

[Dan Santner <da...@me.com>]

I'm about to put together my production environment using couchdb as the backend.  I've been running my test environment on a single linux node (couchdb ver 1.2) for about a year without even restarting it once!  That activity has actually been more than I can imagine in our production environment, however, I'm nervous about going into production running a single node.

So...my question to you guys is this?  Do I look into running big couch, and does that even handle redundancy or just sharding?  Do I simply setup two nodes and let them cross replicate?  Cross replication just seems ripe for problems, but I've never tried it so I'm asking you all what you'd do.

My production traffic will not be high by any measure.  There will be bursts of activity but as mentioned, nothing a single node hasn't been able to handle so far.

Any experiences you guys have to share is appreciated.

Dan.

Re: Best Choice for redundant production environment?

[Alexander Shorin <kx...@gmail.com>]

It's always good to keep somewhere another node that will act as
backup for master one. The backup process could be set via persistent
continuous replications by using replicator db.

But for common case, a bit better to follow "rule of three" in
production : one active master instance, one slave that will become
master when previous one fall and one backup instance that will
restore fallen master without causing additional load on active
instances. Customize scheme to fit your environment, loads and needs.

In my production experience for 2 years of uptime there was never need
in backup/restore process - CouchDB just worked and well, but still
but it's better to be prepared for the worst cases since everything
could happened (hardware problems, bad sectors, etc.).

--
,,,^..^,,,


On Wed, Jul 17, 2013 at 10:41 PM, Dan Santner <da...@me.com> wrote:
> I'm about to put together my production environment using couchdb as the backend.  I've been running my test environment on a single linux node (couchdb ver 1.2) for about a year without even restarting it once!  That activity has actually been more than I can imagine in our production environment, however, I'm nervous about going into production running a single node.
>
> So...my question to you guys is this?  Do I look into running big couch, and does that even handle redundancy or just sharding?  Do I simply setup two nodes and let them cross replicate?  Cross replication just seems ripe for problems, but I've never tried it so I'm asking you all what you'd do.
>
> My production traffic will not be high by any measure.  There will be bursts of activity but as mentioned, nothing a single node hasn't been able to handle so far.
>
> Any experiences you guys have to share is appreciated.
>
> Dan.

Re: Best Choice for redundant production environment?

[Dan Santner <da...@me.com>]

Cool, thanks for the info Adam!   Do you or anyone know what would be the advantage of using more than one port in the range?  Like if I have three nodes perhaps having two in the range allows for quicker communication with the other two than forcing through one port? 

Only other thing that's a bit confusing is....how do I restart bigcouch??  I'm rebooting for now, but would like to know how to just restart bigcouch without doing a kill.

Dan.

On Jul 25, 2013, at 6:33 PM, Adam Kocoloski <ko...@apache.org> wrote:

> Hi Dan, oops, sounds like the docs missed a port range there.  Each of your nodes is listening for connections from the other nodes on a particular port.  That port is advertised by EPMD on 4369.  You can control the range of ports that might be used in the vm.args file -- the following directive placed in that file will force your nodes to listen on port 9100:
> 
>> # Limit the allowable port range for distributed Erlang
>> -kernel inet_dist_listen_min 9100
>> -kernel inet_dist_listen_max 9100
> 
> Regards, Adam
> 
> On Jul 25, 2013, at 4:21 PM, Dan Santner <da...@me.com> wrote:
> 
>> Thanks guys.  After a bit of wrangling I have bigcouch running!  Exciting stuff.  However, does anyone know what ports I'm supposed to open for each node?  
>> The doc only mentions 5984,5986, and 4369 but clearly that wasn't enough.  Started with those three and every PUT gave me a 500 even though it saved to the specific node, just wouldn't propogate to the others..
>> So then I opened all the ports and it worked like a charm.  So clearly there are more ports required than the three in the doc.  Is Erlang speaking over some others?
>> 
>> Sorry if this is the wrong forum.
>> 
>> Dan.
>> 
>> On Jul 17, 2013, at 2:22 PM, Robert Newson <rn...@apache.org> wrote:
>> 
>>> BigCouch provides redundancy and partitioning. By default, three
>>> copies of the data, and uses a quorum mechanism which is generally
>>> superior to three independent couchdb nodes inter-replicating.
>>> 
>>> And, of course, the BigCouch project is merging with CouchDB.
>>> 
>>> B.
>>> 
>>> 
>>> On 17 July 2013 19:59, Nick North <no...@gmail.com> wrote:
>>>> I run a production environment with three nodes several thousand miles
>>>> apart with full-mesh replication - it behaves beautifully, seamlessly
>>>> recovering from network outages, and I don't touch it for months on end. My
>>>> particular setup is multi-master, so every node is a production one, and
>>>> clients swap to another node if their local one goes down. Document ids are
>>>> designed to be globally unique so that they can never clash between nodes,
>>>> and it happens that the app never needs to edit documents, so there is no
>>>> chance of multiple edits colliding.
>>>> 
>>>> Nick
>>>> 
>>>> 
>>>> On 17 July 2013 19:41, Dan Santner <da...@me.com> wrote:
>>>> 
>>>>> I'm about to put together my production environment using couchdb as the
>>>>> backend.  I've been running my test environment on a single linux node
>>>>> (couchdb ver 1.2) for about a year without even restarting it once!  That
>>>>> activity has actually been more than I can imagine in our production
>>>>> environment, however, I'm nervous about going into production running a
>>>>> single node.
>>>>> 
>>>>> So...my question to you guys is this?  Do I look into running big couch,
>>>>> and does that even handle redundancy or just sharding?  Do I simply setup
>>>>> two nodes and let them cross replicate?  Cross replication just seems ripe
>>>>> for problems, but I've never tried it so I'm asking you all what you'd do.
>>>>> 
>>>>> My production traffic will not be high by any measure.  There will be
>>>>> bursts of activity but as mentioned, nothing a single node hasn't been able
>>>>> to handle so far.
>>>>> 
>>>>> Any experiences you guys have to share is appreciated.
>>>>> 
>>>>> Dan.
>> 
> 

Re: Best Choice for redundant production environment?

[Adam Kocoloski <ko...@apache.org>]

Hi Dan, oops, sounds like the docs missed a port range there.  Each of your nodes is listening for connections from the other nodes on a particular port.  That port is advertised by EPMD on 4369.  You can control the range of ports that might be used in the vm.args file -- the following directive placed in that file will force your nodes to listen on port 9100:

> # Limit the allowable port range for distributed Erlang
> -kernel inet_dist_listen_min 9100
> -kernel inet_dist_listen_max 9100

Regards, Adam

On Jul 25, 2013, at 4:21 PM, Dan Santner <da...@me.com> wrote:

> Thanks guys.  After a bit of wrangling I have bigcouch running!  Exciting stuff.  However, does anyone know what ports I'm supposed to open for each node?  
> The doc only mentions 5984,5986, and 4369 but clearly that wasn't enough.  Started with those three and every PUT gave me a 500 even though it saved to the specific node, just wouldn't propogate to the others..
> So then I opened all the ports and it worked like a charm.  So clearly there are more ports required than the three in the doc.  Is Erlang speaking over some others?
> 
> Sorry if this is the wrong forum.
> 
> Dan.
> 
> On Jul 17, 2013, at 2:22 PM, Robert Newson <rn...@apache.org> wrote:
> 
>> BigCouch provides redundancy and partitioning. By default, three
>> copies of the data, and uses a quorum mechanism which is generally
>> superior to three independent couchdb nodes inter-replicating.
>> 
>> And, of course, the BigCouch project is merging with CouchDB.
>> 
>> B.
>> 
>> 
>> On 17 July 2013 19:59, Nick North <no...@gmail.com> wrote:
>>> I run a production environment with three nodes several thousand miles
>>> apart with full-mesh replication - it behaves beautifully, seamlessly
>>> recovering from network outages, and I don't touch it for months on end. My
>>> particular setup is multi-master, so every node is a production one, and
>>> clients swap to another node if their local one goes down. Document ids are
>>> designed to be globally unique so that they can never clash between nodes,
>>> and it happens that the app never needs to edit documents, so there is no
>>> chance of multiple edits colliding.
>>> 
>>> Nick
>>> 
>>> 
>>> On 17 July 2013 19:41, Dan Santner <da...@me.com> wrote:
>>> 
>>>> I'm about to put together my production environment using couchdb as the
>>>> backend.  I've been running my test environment on a single linux node
>>>> (couchdb ver 1.2) for about a year without even restarting it once!  That
>>>> activity has actually been more than I can imagine in our production
>>>> environment, however, I'm nervous about going into production running a
>>>> single node.
>>>> 
>>>> So...my question to you guys is this?  Do I look into running big couch,
>>>> and does that even handle redundancy or just sharding?  Do I simply setup
>>>> two nodes and let them cross replicate?  Cross replication just seems ripe
>>>> for problems, but I've never tried it so I'm asking you all what you'd do.
>>>> 
>>>> My production traffic will not be high by any measure.  There will be
>>>> bursts of activity but as mentioned, nothing a single node hasn't been able
>>>> to handle so far.
>>>> 
>>>> Any experiences you guys have to share is appreciated.
>>>> 
>>>> Dan.
> 

Re: Best Choice for redundant production environment?

[Dan Santner <da...@me.com>]

Thanks guys.  After a bit of wrangling I have bigcouch running!  Exciting stuff.  However, does anyone know what ports I'm supposed to open for each node?  
The doc only mentions 5984,5986, and 4369 but clearly that wasn't enough.  Started with those three and every PUT gave me a 500 even though it saved to the specific node, just wouldn't propogate to the others..
So then I opened all the ports and it worked like a charm.  So clearly there are more ports required than the three in the doc.  Is Erlang speaking over some others?

Sorry if this is the wrong forum.

Dan.

On Jul 17, 2013, at 2:22 PM, Robert Newson <rn...@apache.org> wrote:

> BigCouch provides redundancy and partitioning. By default, three
> copies of the data, and uses a quorum mechanism which is generally
> superior to three independent couchdb nodes inter-replicating.
> 
> And, of course, the BigCouch project is merging with CouchDB.
> 
> B.
> 
> 
> On 17 July 2013 19:59, Nick North <no...@gmail.com> wrote:
>> I run a production environment with three nodes several thousand miles
>> apart with full-mesh replication - it behaves beautifully, seamlessly
>> recovering from network outages, and I don't touch it for months on end. My
>> particular setup is multi-master, so every node is a production one, and
>> clients swap to another node if their local one goes down. Document ids are
>> designed to be globally unique so that they can never clash between nodes,
>> and it happens that the app never needs to edit documents, so there is no
>> chance of multiple edits colliding.
>> 
>> Nick
>> 
>> 
>> On 17 July 2013 19:41, Dan Santner <da...@me.com> wrote:
>> 
>>> I'm about to put together my production environment using couchdb as the
>>> backend.  I've been running my test environment on a single linux node
>>> (couchdb ver 1.2) for about a year without even restarting it once!  That
>>> activity has actually been more than I can imagine in our production
>>> environment, however, I'm nervous about going into production running a
>>> single node.
>>> 
>>> So...my question to you guys is this?  Do I look into running big couch,
>>> and does that even handle redundancy or just sharding?  Do I simply setup
>>> two nodes and let them cross replicate?  Cross replication just seems ripe
>>> for problems, but I've never tried it so I'm asking you all what you'd do.
>>> 
>>> My production traffic will not be high by any measure.  There will be
>>> bursts of activity but as mentioned, nothing a single node hasn't been able
>>> to handle so far.
>>> 
>>> Any experiences you guys have to share is appreciated.
>>> 
>>> Dan.

Re: Best Choice for redundant production environment?

[Robert Newson <rn...@apache.org>]

BigCouch provides redundancy and partitioning. By default, three
copies of the data, and uses a quorum mechanism which is generally
superior to three independent couchdb nodes inter-replicating.

And, of course, the BigCouch project is merging with CouchDB.

B.


On 17 July 2013 19:59, Nick North <no...@gmail.com> wrote:
> I run a production environment with three nodes several thousand miles
> apart with full-mesh replication - it behaves beautifully, seamlessly
> recovering from network outages, and I don't touch it for months on end. My
> particular setup is multi-master, so every node is a production one, and
> clients swap to another node if their local one goes down. Document ids are
> designed to be globally unique so that they can never clash between nodes,
> and it happens that the app never needs to edit documents, so there is no
> chance of multiple edits colliding.
>
> Nick
>
>
> On 17 July 2013 19:41, Dan Santner <da...@me.com> wrote:
>
>> I'm about to put together my production environment using couchdb as the
>> backend.  I've been running my test environment on a single linux node
>> (couchdb ver 1.2) for about a year without even restarting it once!  That
>> activity has actually been more than I can imagine in our production
>> environment, however, I'm nervous about going into production running a
>> single node.
>>
>> So...my question to you guys is this?  Do I look into running big couch,
>> and does that even handle redundancy or just sharding?  Do I simply setup
>> two nodes and let them cross replicate?  Cross replication just seems ripe
>> for problems, but I've never tried it so I'm asking you all what you'd do.
>>
>> My production traffic will not be high by any measure.  There will be
>> bursts of activity but as mentioned, nothing a single node hasn't been able
>> to handle so far.
>>
>> Any experiences you guys have to share is appreciated.
>>
>> Dan.

Re: Best Choice for redundant production environment?

[Nick North <no...@gmail.com>]

I run a production environment with three nodes several thousand miles
apart with full-mesh replication - it behaves beautifully, seamlessly
recovering from network outages, and I don't touch it for months on end. My
particular setup is multi-master, so every node is a production one, and
clients swap to another node if their local one goes down. Document ids are
designed to be globally unique so that they can never clash between nodes,
and it happens that the app never needs to edit documents, so there is no
chance of multiple edits colliding.

Nick


On 17 July 2013 19:41, Dan Santner <da...@me.com> wrote:

> I'm about to put together my production environment using couchdb as the
> backend.  I've been running my test environment on a single linux node
> (couchdb ver 1.2) for about a year without even restarting it once!  That
> activity has actually been more than I can imagine in our production
> environment, however, I'm nervous about going into production running a
> single node.
>
> So...my question to you guys is this?  Do I look into running big couch,
> and does that even handle redundancy or just sharding?  Do I simply setup
> two nodes and let them cross replicate?  Cross replication just seems ripe
> for problems, but I've never tried it so I'm asking you all what you'd do.
>
> My production traffic will not be high by any measure.  There will be
> bursts of activity but as mentioned, nothing a single node hasn't been able
> to handle so far.
>
> Any experiences you guys have to share is appreciated.
>
> Dan.