Re: Partitioning use cases

[Ben Browning <be...@gmail.com>]

David,


On Fri, Feb 27, 2009 at 7:16 PM, David Van Couvering
<da...@vancouvering.com> wrote:
> Hi, all.  I did a first pass at the high-level use cases, both to start with
> and then looking forward.
>
> http://wiki.apache.org/couchdb/Partitioning_proposal

Thanks for adding to the document :)


> I'm sure I'm missing some things (in particular, I don't fully understand
> the discussion about configuring the topology differently depending upon
> performance needs - I wonder if that's an implementation-specific detail).

>From what I understand, the initial idea was to have the partitioning
be fairly static and not try to tackle all the dynamic challenges
up-front. So initially a topology would be decided upon based upon the
needs of a particular dataset (data-intensive, map/reduce intensive,
etc) and remain fairly stable.


> One way perhaps I could help is to work on an API that maps a key to a node
> (with some sort of pluggable interface for various consistent hashing
> algorithms, starting maybe with Chord, given that there is an existing
> implementation in Erlang called Chordial -
> http://dawsdesign.com/drupal/chordial)

I think having a pluggable interface for choosing the consistent
hashing algorithm is a good idea. Chord looks very nice for systems
where nodes are dynamically added and removed all the time. I think
that's a bit more advanced than the initial implementation is shooting
for but maps very well to the long-term goal with CouchDB as a true
distributed database.

These are my thoughts, so if they don't map with the opinions of the
rest of the community please speak up.

Ben

Re: Partitioning use cases

[David Van Couvering <da...@gmail.com>]

> > I'm sure I'm missing some things (in particular, I don't fully understand
> > the discussion about configuring the topology differently depending upon
> > performance needs - I wonder if that's an implementation-specific
> detail).
>
> From what I understand, the initial idea was to have the partitioning
> be fairly static and not try to tackle all the dynamic challenges
> up-front. So initially a topology would be decided upon based upon the
> needs of a particular dataset (data-intensive, map/reduce intensive,
> etc) and remain fairly stable.
>

I guess what I'm trying to understand is this idea of a "tree" topology -
how exactly does that work.  What are the internal nodes of the tree - are
they "meta tables" that contain pointers to the next level of the tree and
or the leaf nodes where the tables contain actual data - ala BigTable?


>
>
> > One way perhaps I could help is to work on an API that maps a key to a
> node
> > (with some sort of pluggable interface for various consistent hashing
> > algorithms, starting maybe with Chord, given that there is an existing
> > implementation in Erlang called Chordial -
> > http://dawsdesign.com/drupal/chordial)
>
> I think having a pluggable interface for choosing the consistent
> hashing algorithm is a good idea. Chord looks very nice for systems
> where nodes are dynamically added and removed all the time. I think
> that's a bit more advanced than the initial implementation is shooting
> for but maps very well to the long-term goal with CouchDB as a true
> distributed database.
>
> These are my thoughts, so if they don't map with the opinions of the
> rest of the community please speak up.
>

OK.  As much as I would like to jump right in, I have a slight problem in
that I am completely new to Erlang.  So I will need to spend some time
coming up to speed with Erlang.  I think I'll approach this in two ways -
walk through the tutorials and then try fixing some bugs.  I can't say when
I will feel capable to build a pluggable interface, but I'll do what I can.

David

>
> Ben
>



-- 
David W. Van Couvering

I am looking for a senior position working on server-side Java systems.
 Feel free to contact me if you know of any opportunities.

http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering