You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Shaun Lindsay <sh...@meebo.com> on 2009/02/20 02:34:38 UTC
Couch clustering/partitioning Re: CouchSpray - Thoughts?
Hi all,
So, a couple months ago we implemented almost exactly the couch
clustering/partitioning solution described below. The couch cluster (which
we called 'The Lounge') sits behind nginx running a custom module that farms
out the GETs and PUTs to the appropriate node/shard and the views to a
python proxy daemon which handles reducing the view results from the
individual shards and returning the full view. We have replication working
between the cluster nodes so the shards exist multiple places and, in the
case of one of the nodes going down, the various proxies fail over to the
backup shards.
This clustering setup has been running in full production for several months
now with minimal problems.
We're looking to release all the code back to the community, but we need to
clear it with our legal team first to make sure we're not compromising any
of our more business-specific, proprietary code.
In total, we have:
a nginx module specifically set up for sharding databases
a 'smartproxy', written in Python/Twisted, for sharding views
and a few other ancillary pieces (replication notification, view updating,
etc)
Mainly, I just wanted to keep people from duplicating the work we've done --
hopefully we can release something back to the community in the next several
weeks.
We're having a meeting tomorrow morning to figure out what we can release
right now (probably the nginx module, at the least). I'll let everyone know
what out timeline looks like.
--Shaun Lindsay
Meebo.com
On Thu, Feb 19, 2009 at 4:48 PM, Chris Anderson <jc...@apache.org> wrote:
> On Thu, Feb 19, 2009 at 4:35 PM, Ben Browning <be...@gmail.com> wrote:
> > So, I started thinking about partitioning with CouchDB and realized
> > that since views are just map/reduce, we can do some magic that's
> > harder if not impossible with other database systems. The idea in a
> > nutshell is to create a proxy that sits in front of multiple servers
> > and "sprays" the view queries to all servers, merging the results -
> > hence CouchSpray. This would give us storage and processing
> > scalability and could, with some extra logic, provide data redundancy
> > and failover.
>
> There are plans in CouchDB's future to take care of data partitioning,
> as well as querying views from a cluster. Theoretically, it should be
> pretty simple. There are a few small projects that have started down
> the road of writing code in this area.
>
> https://code.launchpad.net/~dreid/sectional/trunk
>
> Sectional is an Erlang http proxy that implements consistent hashing
> for docs. I'm not sure how it handles view queries.
>
> There's also a project to provide partitioning around the basic
> key/value PUT and GET store using Nginx:
>
> http://github.com/dysinger/nginx/tree/nginx_upstream_hash
>
> If you're interested in digging into this stuff, please join dev@. We
> plan to include clustering in CouchDB, so if you're interested in
> implementing it, we could use your help.
>
> Chris
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Ben Browning <be...@gmail.com>.
I'd be very interested in any code that you could release. If that's
not possible then
your experiences and challenges faced while implementing this cluster would
definitely be a great help to the community.
It's encouraging to see someone with a working cluster already running.
Ben
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Shaun Lindsay <sh...@meebo.com>.
Our current cluster is running on 4 nodes, on some slower, leftover
hardware. As of right now, it's handling about 300 queries/sec, with about
1/3 of that being view requests. As for documents, we're looking at ~12M
docs taking up around 90Gb of space on disk.
We have the cluster split in to 48 shards, with 12 shards per node -- this
lets us add more nodes later, up to 48, before we need to mess with
resharding the data (or, more likely, treeing out underneath the first level
of nodes).
We've also implemented some caching for the most often hit views in the
smartproxy -- map/reducing over the 48 shards isn't trivial and, at 100 view
queries a second, caching is necessary for the cluster to actually work.
--Shaun Lindsay
Meebo.com
On Thu, Feb 19, 2009 at 5:40 PM, Chris Anderson <jc...@apache.org> wrote:
> On Thu, Feb 19, 2009 at 5:34 PM, Shaun Lindsay <sh...@meebo.com> wrote:
> > Hi all,
> > So, a couple months ago we implemented almost exactly the couch
> > clustering/partitioning solution described below. The couch cluster
> (which
> > we called 'The Lounge') sits behind nginx running a custom module that
> farms
> > out the GETs and PUTs to the appropriate node/shard and the views to a
> > python proxy daemon which handles reducing the view results from the
> > individual shards and returning the full view. We have replication
> working
> > between the cluster nodes so the shards exist multiple places and, in the
> > case of one of the nodes going down, the various proxies fail over to the
> > backup shards.
> >
> > This clustering setup has been running in full production for several
> months
> > now with minimal problems.
> >
> > We're looking to release all the code back to the community, but we need
> to
> > clear it with our legal team first to make sure we're not compromising
> any
> > of our more business-specific, proprietary code.
> >
> > In total, we have:
> > a nginx module specifically set up for sharding databases
> > a 'smartproxy', written in Python/Twisted, for sharding views
> > and a few other ancillary pieces (replication notification, view
> updating,
> > etc)
> >
> > Mainly, I just wanted to keep people from duplicating the work we've done
> --
> > hopefully we can release something back to the community in the next
> several
> > weeks.
> >
> > We're having a meeting tomorrow morning to figure out what we can release
> > right now (probably the nginx module, at the least). I'll let everyone
> know
> > what out timeline looks like.
> >
>
> This is really cool. It sounds like a tool people could get started
> with right away to build big CouchDB clusters. Is there anything you
> can tell us about the size of clusters you've used?
>
> The smartproxy code will probably be a really good illustration of
> what we'll probably want to implement in Erlang for CouchDB,
> eventually. Again, +1 for software people can use today.
>
> Hopefully it turns out to be easy to release!
>
> Cheers,
> Chris
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Shaun Lindsay <sh...@meebo.com>.
Our current cluster is running on 4 nodes, on some slower, leftover
hardware. As of right now, it's handling about 300 queries/sec, with about
1/3 of that being view requests. As for documents, we're looking at ~12M
docs taking up around 90Gb of space on disk.
We have the cluster split in to 48 shards, with 12 shards per node -- this
lets us add more nodes later, up to 48, before we need to mess with
resharding the data (or, more likely, treeing out underneath the first level
of nodes).
We've also implemented some caching for the most often hit views in the
smartproxy -- map/reducing over the 48 shards isn't trivial and, at 100 view
queries a second, caching is necessary for the cluster to actually work.
--Shaun Lindsay
Meebo.com
On Thu, Feb 19, 2009 at 5:40 PM, Chris Anderson <jc...@apache.org> wrote:
> On Thu, Feb 19, 2009 at 5:34 PM, Shaun Lindsay <sh...@meebo.com> wrote:
> > Hi all,
> > So, a couple months ago we implemented almost exactly the couch
> > clustering/partitioning solution described below. The couch cluster
> (which
> > we called 'The Lounge') sits behind nginx running a custom module that
> farms
> > out the GETs and PUTs to the appropriate node/shard and the views to a
> > python proxy daemon which handles reducing the view results from the
> > individual shards and returning the full view. We have replication
> working
> > between the cluster nodes so the shards exist multiple places and, in the
> > case of one of the nodes going down, the various proxies fail over to the
> > backup shards.
> >
> > This clustering setup has been running in full production for several
> months
> > now with minimal problems.
> >
> > We're looking to release all the code back to the community, but we need
> to
> > clear it with our legal team first to make sure we're not compromising
> any
> > of our more business-specific, proprietary code.
> >
> > In total, we have:
> > a nginx module specifically set up for sharding databases
> > a 'smartproxy', written in Python/Twisted, for sharding views
> > and a few other ancillary pieces (replication notification, view
> updating,
> > etc)
> >
> > Mainly, I just wanted to keep people from duplicating the work we've done
> --
> > hopefully we can release something back to the community in the next
> several
> > weeks.
> >
> > We're having a meeting tomorrow morning to figure out what we can release
> > right now (probably the nginx module, at the least). I'll let everyone
> know
> > what out timeline looks like.
> >
>
> This is really cool. It sounds like a tool people could get started
> with right away to build big CouchDB clusters. Is there anything you
> can tell us about the size of clusters you've used?
>
> The smartproxy code will probably be a really good illustration of
> what we'll probably want to implement in Erlang for CouchDB,
> eventually. Again, +1 for software people can use today.
>
> Hopefully it turns out to be easy to release!
>
> Cheers,
> Chris
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Feb 19, 2009 at 5:34 PM, Shaun Lindsay <sh...@meebo.com> wrote:
> Hi all,
> So, a couple months ago we implemented almost exactly the couch
> clustering/partitioning solution described below. The couch cluster (which
> we called 'The Lounge') sits behind nginx running a custom module that farms
> out the GETs and PUTs to the appropriate node/shard and the views to a
> python proxy daemon which handles reducing the view results from the
> individual shards and returning the full view. We have replication working
> between the cluster nodes so the shards exist multiple places and, in the
> case of one of the nodes going down, the various proxies fail over to the
> backup shards.
>
> This clustering setup has been running in full production for several months
> now with minimal problems.
>
> We're looking to release all the code back to the community, but we need to
> clear it with our legal team first to make sure we're not compromising any
> of our more business-specific, proprietary code.
>
> In total, we have:
> a nginx module specifically set up for sharding databases
> a 'smartproxy', written in Python/Twisted, for sharding views
> and a few other ancillary pieces (replication notification, view updating,
> etc)
>
> Mainly, I just wanted to keep people from duplicating the work we've done --
> hopefully we can release something back to the community in the next several
> weeks.
>
> We're having a meeting tomorrow morning to figure out what we can release
> right now (probably the nginx module, at the least). I'll let everyone know
> what out timeline looks like.
>
This is really cool. It sounds like a tool people could get started
with right away to build big CouchDB clusters. Is there anything you
can tell us about the size of clusters you've used?
The smartproxy code will probably be a really good illustration of
what we'll probably want to implement in Erlang for CouchDB,
eventually. Again, +1 for software people can use today.
Hopefully it turns out to be easy to release!
Cheers,
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Feb 19, 2009 at 5:34 PM, Shaun Lindsay <sh...@meebo.com> wrote:
> Hi all,
> So, a couple months ago we implemented almost exactly the couch
> clustering/partitioning solution described below. The couch cluster (which
> we called 'The Lounge') sits behind nginx running a custom module that farms
> out the GETs and PUTs to the appropriate node/shard and the views to a
> python proxy daemon which handles reducing the view results from the
> individual shards and returning the full view. We have replication working
> between the cluster nodes so the shards exist multiple places and, in the
> case of one of the nodes going down, the various proxies fail over to the
> backup shards.
>
> This clustering setup has been running in full production for several months
> now with minimal problems.
>
> We're looking to release all the code back to the community, but we need to
> clear it with our legal team first to make sure we're not compromising any
> of our more business-specific, proprietary code.
>
> In total, we have:
> a nginx module specifically set up for sharding databases
> a 'smartproxy', written in Python/Twisted, for sharding views
> and a few other ancillary pieces (replication notification, view updating,
> etc)
>
> Mainly, I just wanted to keep people from duplicating the work we've done --
> hopefully we can release something back to the community in the next several
> weeks.
>
> We're having a meeting tomorrow morning to figure out what we can release
> right now (probably the nginx module, at the least). I'll let everyone know
> what out timeline looks like.
>
This is really cool. It sounds like a tool people could get started
with right away to build big CouchDB clusters. Is there anything you
can tell us about the size of clusters you've used?
The smartproxy code will probably be a really good illustration of
what we'll probably want to implement in Erlang for CouchDB,
eventually. Again, +1 for software people can use today.
Hopefully it turns out to be easy to release!
Cheers,
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Chris Anderson <jc...@apache.org>.
On Fri, Feb 20, 2009 at 4:53 PM, Shaun Lindsay <sh...@meebo.com> wrote:
> Due to one of the key people being sick, we pushed our meeting to discuss
> releasing the code to Monday. I'll send out an update then.
>
Thanks for keeping us in the loop!
Hope you all are feeling better.
--
Chris Anderson
http://jchris.mfdz.com
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Shaun Lindsay <sh...@meebo.com>.
Due to one of the key people being sick, we pushed our meeting to discuss
releasing the code to Monday. I'll send out an update then.
On Fri, Feb 20, 2009 at 2:17 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 20 Feb 2009, at 02:34, Shaun Lindsay wrote:
>
> Hi all,
>> So, a couple months ago we implemented almost exactly the couch
>> clustering/partitioning solution described below.
>>
>
> Shaun, this sounds fantastic! :) I hope you can release the code for
> this.
>
> Cheers
> Jan
> --
>
>
>
>
> The couch cluster (which
>> we called 'The Lounge') sits behind nginx running a custom module that
>> farms
>> out the GETs and PUTs to the appropriate node/shard and the views to a
>> python proxy daemon which handles reducing the view results from the
>> individual shards and returning the full view. We have replication
>> working
>> between the cluster nodes so the shards exist multiple places and, in the
>> case of one of the nodes going down, the various proxies fail over to the
>> backup shards.
>>
>> This clustering setup has been running in full production for several
>> months
>> now with minimal problems.
>>
>> We're looking to release all the code back to the community, but we need
>> to
>> clear it with our legal team first to make sure we're not compromising any
>> of our more business-specific, proprietary code.
>>
>> In total, we have:
>> a nginx module specifically set up for sharding databases
>> a 'smartproxy', written in Python/Twisted, for sharding views
>> and a few other ancillary pieces (replication notification, view updating,
>> etc)
>>
>> Mainly, I just wanted to keep people from duplicating the work we've done
>> --
>> hopefully we can release something back to the community in the next
>> several
>> weeks.
>>
>> We're having a meeting tomorrow morning to figure out what we can release
>> right now (probably the nginx module, at the least). I'll let everyone
>> know
>> what out timeline looks like.
>>
>> --Shaun Lindsay
>> Meebo.com
>>
>> On Thu, Feb 19, 2009 at 4:48 PM, Chris Anderson <jc...@apache.org>
>> wrote:
>>
>> On Thu, Feb 19, 2009 at 4:35 PM, Ben Browning <be...@gmail.com> wrote:
>>>
>>>> So, I started thinking about partitioning with CouchDB and realized
>>>> that since views are just map/reduce, we can do some magic that's
>>>> harder if not impossible with other database systems. The idea in a
>>>> nutshell is to create a proxy that sits in front of multiple servers
>>>> and "sprays" the view queries to all servers, merging the results -
>>>> hence CouchSpray. This would give us storage and processing
>>>> scalability and could, with some extra logic, provide data redundancy
>>>> and failover.
>>>>
>>>
>>> There are plans in CouchDB's future to take care of data partitioning,
>>> as well as querying views from a cluster. Theoretically, it should be
>>> pretty simple. There are a few small projects that have started down
>>> the road of writing code in this area.
>>>
>>> https://code.launchpad.net/~dreid/sectional/trunk<https://code.launchpad.net/%7Edreid/sectional/trunk>
>>>
>>> Sectional is an Erlang http proxy that implements consistent hashing
>>> for docs. I'm not sure how it handles view queries.
>>>
>>> There's also a project to provide partitioning around the basic
>>> key/value PUT and GET store using Nginx:
>>>
>>> http://github.com/dysinger/nginx/tree/nginx_upstream_hash
>>>
>>> If you're interested in digging into this stuff, please join dev@. We
>>> plan to include clustering in CouchDB, so if you're interested in
>>> implementing it, we could use your help.
>>>
>>> Chris
>>>
>>> --
>>> Chris Anderson
>>> http://jchris.mfdz.com
>>>
>>>
>
Re: Couch clustering/partitioning Re: CouchSpray - Thoughts?
Posted by Jan Lehnardt <ja...@apache.org>.
On 20 Feb 2009, at 02:34, Shaun Lindsay wrote:
> Hi all,
> So, a couple months ago we implemented almost exactly the couch
> clustering/partitioning solution described below.
Shaun, this sounds fantastic! :) I hope you can release the code for
this.
Cheers
Jan
--
> The couch cluster (which
> we called 'The Lounge') sits behind nginx running a custom module
> that farms
> out the GETs and PUTs to the appropriate node/shard and the views to a
> python proxy daemon which handles reducing the view results from the
> individual shards and returning the full view. We have replication
> working
> between the cluster nodes so the shards exist multiple places and,
> in the
> case of one of the nodes going down, the various proxies fail over
> to the
> backup shards.
>
> This clustering setup has been running in full production for
> several months
> now with minimal problems.
>
> We're looking to release all the code back to the community, but we
> need to
> clear it with our legal team first to make sure we're not
> compromising any
> of our more business-specific, proprietary code.
>
> In total, we have:
> a nginx module specifically set up for sharding databases
> a 'smartproxy', written in Python/Twisted, for sharding views
> and a few other ancillary pieces (replication notification, view
> updating,
> etc)
>
> Mainly, I just wanted to keep people from duplicating the work we've
> done --
> hopefully we can release something back to the community in the next
> several
> weeks.
>
> We're having a meeting tomorrow morning to figure out what we can
> release
> right now (probably the nginx module, at the least). I'll let
> everyone know
> what out timeline looks like.
>
> --Shaun Lindsay
> Meebo.com
>
> On Thu, Feb 19, 2009 at 4:48 PM, Chris Anderson <jc...@apache.org>
> wrote:
>
>> On Thu, Feb 19, 2009 at 4:35 PM, Ben Browning <be...@gmail.com>
>> wrote:
>>> So, I started thinking about partitioning with CouchDB and realized
>>> that since views are just map/reduce, we can do some magic that's
>>> harder if not impossible with other database systems. The idea in a
>>> nutshell is to create a proxy that sits in front of multiple servers
>>> and "sprays" the view queries to all servers, merging the results -
>>> hence CouchSpray. This would give us storage and processing
>>> scalability and could, with some extra logic, provide data
>>> redundancy
>>> and failover.
>>
>> There are plans in CouchDB's future to take care of data
>> partitioning,
>> as well as querying views from a cluster. Theoretically, it should be
>> pretty simple. There are a few small projects that have started down
>> the road of writing code in this area.
>>
>> https://code.launchpad.net/~dreid/sectional/trunk
>>
>> Sectional is an Erlang http proxy that implements consistent hashing
>> for docs. I'm not sure how it handles view queries.
>>
>> There's also a project to provide partitioning around the basic
>> key/value PUT and GET store using Nginx:
>>
>> http://github.com/dysinger/nginx/tree/nginx_upstream_hash
>>
>> If you're interested in digging into this stuff, please join dev@. We
>> plan to include clustering in CouchDB, so if you're interested in
>> implementing it, we could use your help.
>>
>> Chris
>>
>> --
>> Chris Anderson
>> http://jchris.mfdz.com
>>