You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by niall el-assaad <ni...@gmail.com> on 2011/02/28 01:13:39 UTC
How many nodes can couchdb scale to?
Hi,
I'm looking at developing an application that will have a couple of central
nodes (data centre) and around 2,000 remote nodes (in branch offices).
I'd like to know if couchdb can scale to having this many nodes working in a
cluster.
I've looked through all the docs, but can't seem to see if there are any
limits or best practises.
thanks,
Re: How many nodes can couchdb scale to?
Posted by niall el-assaad <ni...@gmail.com>.
Thanks Isaac, thats a great idea makes sense, it would depend on the network
topology, but for MPLS networks there would be no additional traffic load.
On Tue, Mar 1, 2011 at 12:33 AM, Isaac Force <is...@autognosis.org> wrote:
> >> Hi Niall, I think the key part is that with this topology your central
> >> servers are going to need to support a sustained throughput of 20,000
> >> reads/second in order to distribute the updates to all 2,000 servers.
> >> Granted, each read is repeated 2,000 times, so you'll mostly be reading
> >> from page cache, but a cached read from CouchDB is not nearly as cheap
> as
> >> reading from e.g. Varnish.
> >
> > Thanks Adam,
> >
> > Thats a good point. I suppose we could scale this by adding more nodes in
> > the data centre.
>
> An alternative would be to create peer-to-peer replication rings from
> your edge nodes with a limited set of replication 'uplinks' to the
> data center: draw nodes in an even ring, connect each node to its
> adjacent peer to make a circle, connect each node to the node opposite
> it in the ring, connect a set of nodes relative to the size of the
> ring to the data center.
>
> If a set of nodes die, each node should still have an intra-ring path
> to each remaining node, and as long as there's still one uplink
> remaining replication from the data center will continue. You could
> even connect the rings to one another and have the data center merely
> be another node on one of the rings.
>
> This way configuration complexity and burden at the edge is traded for
> egress replication burden from the data center.
>
> -Isaac
>
Re: How many nodes can couchdb scale to?
Posted by Isaac Force <is...@autognosis.org>.
>> Hi Niall, I think the key part is that with this topology your central
>> servers are going to need to support a sustained throughput of 20,000
>> reads/second in order to distribute the updates to all 2,000 servers.
>> Granted, each read is repeated 2,000 times, so you'll mostly be reading
>> from page cache, but a cached read from CouchDB is not nearly as cheap as
>> reading from e.g. Varnish.
>
> Thanks Adam,
>
> Thats a good point. I suppose we could scale this by adding more nodes in
> the data centre.
An alternative would be to create peer-to-peer replication rings from
your edge nodes with a limited set of replication 'uplinks' to the
data center: draw nodes in an even ring, connect each node to its
adjacent peer to make a circle, connect each node to the node opposite
it in the ring, connect a set of nodes relative to the size of the
ring to the data center.
If a set of nodes die, each node should still have an intra-ring path
to each remaining node, and as long as there's still one uplink
remaining replication from the data center will continue. You could
even connect the rings to one another and have the data center merely
be another node on one of the rings.
This way configuration complexity and burden at the edge is traded for
egress replication burden from the data center.
-Isaac
Re: How many nodes can couchdb scale to?
Posted by niall el-assaad <ni...@gmail.com>.
Thanks Adam,
Thats a good point. I suppose we could scale this by adding more nodes in
the data centre.
> Hi Niall, I think the key part is that with this topology your central
> servers are going to need to support a sustained throughput of 20,000
> reads/second in order to distribute the updates to all 2,000 servers.
> Granted, each read is repeated 2,000 times, so you'll mostly be reading
> from page cache, but a cached read from CouchDB is not nearly as cheap as
> reading from e.g. Varnish.
>
> Adam
Re: How many nodes can couchdb scale to?
Posted by Adam Kocoloski <ko...@apache.org>.
On Feb 28, 2011, at 3:39 AM, niall el-assaad wrote:
> Hi Isaac,
>
> Thanks for the reply. I've put some more info inline:
>
>>
>>
>> * Replication topology: is the plan to have replication from the branch
>> office
>> nodes to your centralized data center? (n:1)
>>
>
> We would have a single box at each branch office replicating with two boxes
> at the data centre (for resilience).
> The data centre boxes would then replicate with each other.
>
> So if some data was inserted at the remote branch, it would be replicated to
> the data centre, and the data centre would replicate it to the other 1999
> branches.
>
>
>> * Replication type: continuous or triggered manually/programatically?
>>
>
> The ideal would be continuous.
>
>
>> * Scope of data set: I would be more concerned with writes than reads.
>> You'll need to have an idea of what your current aggregate average and
>> peak writes per second are, how much data is written for a given
>> period of time, and how far you think you will need this rate to scale in
>> the future.
>>
>
> I would expect the average to be around 10 writes per second, with the peak
> at about 100 writes per second.
>
>
>> * Why Couch: is CouchDB going to be addressing a brand new need, or
>> is it going to replace existing systems for known reasons? If it's the
>> latter, what is it about your current systems that aren't meeting your
>> demands, and what do you hope Couch will provide that will fill the gap?
>> (Specifically looking for performance data that you might have already
>> collected, and if Couch is going to be living on your existing hardware
>> or new hardware.)
>>
>
> Its for a completely new project, the main driver for looking at CouchDB is
> the ability to have a very large scale cluster with write capabilities in
> each branch. Mainly so if their is a failure to communications between the
> branch and the data centre everything can continue to work, then sync up
> later.
>
>>
>> I haven't dealt with large distributed Couch systems, but my instinct
>> would be that Couch wouldn't have any problem with a 2000:1 replicated
>> system. (See Ubuntu One as an example of a large CouchDB system with
>> many external replicators.) The ability to handle it would come down
>> to how well the aggregate data set matches the size of hardware and
>> replication layout in your data center, and of course available
>> ingress bandwidth.
>>
>
> Understand, we would scale the hardware and bandwidth accordingly based upon
> testing of the application.
>
> thanks,
>
> niall
Hi Niall, I think the key part is that with this topology your central servers are going to need to support a sustained throughput of 20,000 reads/second in order to distribute the updates to all 2,000 servers. Granted, each read is repeated 2,000 times, so you'll mostly be reading from page cache, but a cached read from CouchDB is not nearly as cheap as reading from e.g. Varnish.
Adam
Re: How many nodes can couchdb scale to?
Posted by niall el-assaad <ni...@gmail.com>.
Hi Isaac,
Thanks for the reply. I've put some more info inline:
>
>
> * Replication topology: is the plan to have replication from the branch
> office
> nodes to your centralized data center? (n:1)
>
We would have a single box at each branch office replicating with two boxes
at the data centre (for resilience).
The data centre boxes would then replicate with each other.
So if some data was inserted at the remote branch, it would be replicated to
the data centre, and the data centre would replicate it to the other 1999
branches.
> * Replication type: continuous or triggered manually/programatically?
>
The ideal would be continuous.
> * Scope of data set: I would be more concerned with writes than reads.
> You'll need to have an idea of what your current aggregate average and
> peak writes per second are, how much data is written for a given
> period of time, and how far you think you will need this rate to scale in
> the future.
>
I would expect the average to be around 10 writes per second, with the peak
at about 100 writes per second.
> * Why Couch: is CouchDB going to be addressing a brand new need, or
> is it going to replace existing systems for known reasons? If it's the
> latter, what is it about your current systems that aren't meeting your
> demands, and what do you hope Couch will provide that will fill the gap?
> (Specifically looking for performance data that you might have already
> collected, and if Couch is going to be living on your existing hardware
> or new hardware.)
>
Its for a completely new project, the main driver for looking at CouchDB is
the ability to have a very large scale cluster with write capabilities in
each branch. Mainly so if their is a failure to communications between the
branch and the data centre everything can continue to work, then sync up
later.
>
> I haven't dealt with large distributed Couch systems, but my instinct
> would be that Couch wouldn't have any problem with a 2000:1 replicated
> system. (See Ubuntu One as an example of a large CouchDB system with
> many external replicators.) The ability to handle it would come down
> to how well the aggregate data set matches the size of hardware and
> replication layout in your data center, and of course available
> ingress bandwidth.
>
Understand, we would scale the hardware and bandwidth accordingly based upon
testing of the application.
thanks,
niall
Re: How many nodes can couchdb scale to?
Posted by David Beckwith <th...@gmail.com>.
Sounds like fun. Let me know if you guys need help. I might be able
to contribute a few nodes for the test.
D :)
On Sun, Feb 27, 2011 at 6:20 PM, Adam Kocoloski <ko...@apache.org> wrote:
> On Feb 27, 2011, at 8:56 PM, Isaac Force wrote:
>
>> On Sun, Feb 27, 2011 at 4:13 PM, niall el-assaad <ni...@gmail.com> wrote:
>>> I'm looking at developing an application that will have a couple of central
>>> nodes (data centre) and around 2,000 remote nodes (in branch offices).
>>>
>>> I'd like to know if couchdb can scale to having this many nodes working in a
>>> cluster.
>>
>> For the list to write a more meaningful answer to address your needs,
>> it would be helpful if you gave more detail about the nature of the
>> problem you want to solve.
>>
>> * Replication topology: is the plan to have replication from the branch office
>> nodes to your centralized data center? (n:1)
>> * Replication type: continuous or triggered manually/programatically?
>> * Scope of data set: I would be more concerned with writes than reads.
>> You'll need to have an idea of what your current aggregate average and
>> peak writes per second are, how much data is written for a given
>> period of time, and how far you think you will need this rate to scale in
>> the future.
>> * Why Couch: is CouchDB going to be addressing a brand new need, or
>> is it going to replace existing systems for known reasons? If it's the
>> latter, what is it about your current systems that aren't meeting your
>> demands, and what do you hope Couch will provide that will fill the gap?
>> (Specifically looking for performance data that you might have already
>> collected, and if Couch is going to be living on your existing hardware
>> or new hardware.)
>>
>> I haven't dealt with large distributed Couch systems, but my instinct
>> would be that Couch wouldn't have any problem with a 2000:1 replicated
>> system. (See Ubuntu One as an example of a large CouchDB system with
>> many external replicators.) The ability to handle it would come down
>> to how well the aggregate data set matches the size of hardware and
>> replication layout in your data center, and of course available
>> ingress bandwidth.
>>
>> -Isaac
>
> Well said Isaac.
>
>
Re: How many nodes can couchdb scale to?
Posted by Adam Kocoloski <ko...@apache.org>.
On Feb 27, 2011, at 8:56 PM, Isaac Force wrote:
> On Sun, Feb 27, 2011 at 4:13 PM, niall el-assaad <ni...@gmail.com> wrote:
>> I'm looking at developing an application that will have a couple of central
>> nodes (data centre) and around 2,000 remote nodes (in branch offices).
>>
>> I'd like to know if couchdb can scale to having this many nodes working in a
>> cluster.
>
> For the list to write a more meaningful answer to address your needs,
> it would be helpful if you gave more detail about the nature of the
> problem you want to solve.
>
> * Replication topology: is the plan to have replication from the branch office
> nodes to your centralized data center? (n:1)
> * Replication type: continuous or triggered manually/programatically?
> * Scope of data set: I would be more concerned with writes than reads.
> You'll need to have an idea of what your current aggregate average and
> peak writes per second are, how much data is written for a given
> period of time, and how far you think you will need this rate to scale in
> the future.
> * Why Couch: is CouchDB going to be addressing a brand new need, or
> is it going to replace existing systems for known reasons? If it's the
> latter, what is it about your current systems that aren't meeting your
> demands, and what do you hope Couch will provide that will fill the gap?
> (Specifically looking for performance data that you might have already
> collected, and if Couch is going to be living on your existing hardware
> or new hardware.)
>
> I haven't dealt with large distributed Couch systems, but my instinct
> would be that Couch wouldn't have any problem with a 2000:1 replicated
> system. (See Ubuntu One as an example of a large CouchDB system with
> many external replicators.) The ability to handle it would come down
> to how well the aggregate data set matches the size of hardware and
> replication layout in your data center, and of course available
> ingress bandwidth.
>
> -Isaac
Well said Isaac.
Re: How many nodes can couchdb scale to?
Posted by Isaac Force <is...@autognosis.org>.
On Sun, Feb 27, 2011 at 4:13 PM, niall el-assaad <ni...@gmail.com> wrote:
> I'm looking at developing an application that will have a couple of central
> nodes (data centre) and around 2,000 remote nodes (in branch offices).
>
> I'd like to know if couchdb can scale to having this many nodes working in a
> cluster.
For the list to write a more meaningful answer to address your needs,
it would be helpful if you gave more detail about the nature of the
problem you want to solve.
* Replication topology: is the plan to have replication from the branch office
nodes to your centralized data center? (n:1)
* Replication type: continuous or triggered manually/programatically?
* Scope of data set: I would be more concerned with writes than reads.
You'll need to have an idea of what your current aggregate average and
peak writes per second are, how much data is written for a given
period of time, and how far you think you will need this rate to scale in
the future.
* Why Couch: is CouchDB going to be addressing a brand new need, or
is it going to replace existing systems for known reasons? If it's the
latter, what is it about your current systems that aren't meeting your
demands, and what do you hope Couch will provide that will fill the gap?
(Specifically looking for performance data that you might have already
collected, and if Couch is going to be living on your existing hardware
or new hardware.)
I haven't dealt with large distributed Couch systems, but my instinct
would be that Couch wouldn't have any problem with a 2000:1 replicated
system. (See Ubuntu One as an example of a large CouchDB system with
many external replicators.) The ability to handle it would come down
to how well the aggregate data set matches the size of hardware and
replication layout in your data center, and of course available
ingress bandwidth.
-Isaac
Re: How many nodes can couchdb scale to?
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 28 February 2011 13:13, niall el-assaad <ni...@gmail.com> wrote:
> Hi,
>
> I'm looking at developing an application that will have a couple of central
> nodes (data centre) and around 2,000 remote nodes (in branch offices).
>
> I'd like to know if couchdb can scale to having this many nodes working in a
> cluster.
>
> I've looked through all the docs, but can't seem to see if there are any
> limits or best practises.
>
> thanks,
>
I'm also curious. If we have no benchmarks, I am keen to contribute to
a test for this (NZ is pretty much like a remote office ...)
Is anybody interested in helping organise a mega couch testing
hackathon? sometime in the next month? If so what parameters/info
would we want to test & to collect?
A+
Dave