CouchSpray - Thoughts?

[Ben Browning <be...@gmail.com>]

So, I started thinking about partitioning with CouchDB and realized
that since views are just map/reduce, we can do some magic that's
harder if not impossible with other database systems. The idea in a
nutshell is to create a proxy that sits in front of multiple servers
and "sprays" the view queries to all servers, merging the results -
hence CouchSpray. This would give us storage and processing
scalability and could, with some extra logic, provide data redundancy
and failover.

I have a github repo that as of right now only contains a README file
with more details:

http://github.com/bbrowning/couchspray/tree/master

I'd like to open the discussion up to the community before starting on
the code. Does this sound useful? Is someone else already working on a
similar solution that I could contribute to instead of duplicating
effort? My initial idea would be to use mochiweb as the proxy in a
manner similar to fuzed
(http://github.com/KirinDave/fuzed/tree/master) and, license
permitting, potentially borrow some code from that project as well as
re-use some of Couch's existing API handling and map/reduce code.

Thanks,

- Ben

Re: CouchSpray - Thoughts?

[Chris Anderson <jc...@apache.org>]

On Thu, Feb 19, 2009 at 5:10 PM, Ben Browning <be...@gmail.com> wrote:
> On Thu, Feb 19, 2009 at 7:48 PM, Chris Anderson <jc...@apache.org> wrote:
>> If you're interested in digging into this stuff, please join dev@. We
>> plan to include clustering in CouchDB, so if you're interested in
>> implementing it, we could use your help.
>
> I've been on dev@ since the beginning but until this point I've just
> been a lurker. I'm definitely up for helping. The only reason I didn't
> suggest helping with this in Couch directly is that will require much
> more input from the Couch devs and I didn't know how much internal
> code churn would be taking place before the 1.0 release.
>
> I'll start a separate thread over there to get things moving.
>

I'm in the middle of writing up an overview of what's been informally
discussed (tree of nodes model). I'll post that to dev tonight also.

Chris


-- 
Chris Anderson
http://jchris.mfdz.com

Re: CouchSpray - Thoughts?

[Ben Browning <be...@gmail.com>]

On Thu, Feb 19, 2009 at 7:48 PM, Chris Anderson <jc...@apache.org> wrote:
> If you're interested in digging into this stuff, please join dev@. We
> plan to include clustering in CouchDB, so if you're interested in
> implementing it, we could use your help.

I've been on dev@ since the beginning but until this point I've just
been a lurker. I'm definitely up for helping. The only reason I didn't
suggest helping with this in Couch directly is that will require much
more input from the Couch devs and I didn't know how much internal
code churn would be taking place before the 1.0 release.

I'll start a separate thread over there to get things moving.

- Ben

Re: CouchSpray - Thoughts?

[Chris Anderson <jc...@apache.org>]

On Thu, Feb 19, 2009 at 4:35 PM, Ben Browning <be...@gmail.com> wrote:
> So, I started thinking about partitioning with CouchDB and realized
> that since views are just map/reduce, we can do some magic that's
> harder if not impossible with other database systems. The idea in a
> nutshell is to create a proxy that sits in front of multiple servers
> and "sprays" the view queries to all servers, merging the results -
> hence CouchSpray. This would give us storage and processing
> scalability and could, with some extra logic, provide data redundancy
> and failover.

There are plans in CouchDB's future to take care of data partitioning,
as well as querying views from a cluster. Theoretically, it should be
pretty simple. There are a few small projects that have started down
the road of writing code in this area.

https://code.launchpad.net/~dreid/sectional/trunk

Sectional is an Erlang http proxy that implements consistent hashing
for docs. I'm not sure how it handles view queries.

There's also a project to provide partitioning around the basic
key/value PUT and GET store using Nginx:

http://github.com/dysinger/nginx/tree/nginx_upstream_hash

If you're interested in digging into this stuff, please join dev@. We
plan to include clustering in CouchDB, so if you're interested in
implementing it, we could use your help.

Chris

-- 
Chris Anderson
http://jchris.mfdz.com