Moving Partitions

[Stefan Podkowinski <sp...@gmail.com>]

Hello Couch Users

As I'm about to develop a deployment strategy for my current company,
theres one aspect left that keeps me puzzled. While couch seems to be
pretty disk hungry I need a way to scale in terms of disk space and
decided to setup oversharding right from the start. I plan to create a
couple of logical partitions on a single server that I can move later
to another server to free disk space. Propably using lounge for that.

What I'm wondering though is how to move the data to a new node in
that case. E.g. lets say I have node A with 16 logical partitions. Now
I like to move 8 of them to a new node B. How can I move the data for
only those 8 partitions to the new node? I can't just reimport the
whole data as that would take too much time. Is that even possible to
split an existing node that way?

Cheers,
Stefan

Re: Moving Partitions

[Robert Dionne <di...@dionne-associates.com>]

You may want to look at bigcouch[1] also.

[1] https://github.com/cloudant/bigcouch




On May 2, 2011, at 6:43 AM, Stefan Podkowinski wrote:

> Hello Couch Users
> 
> As I'm about to develop a deployment strategy for my current company,
> theres one aspect left that keeps me puzzled. While couch seems to be
> pretty disk hungry I need a way to scale in terms of disk space and
> decided to setup oversharding right from the start. I plan to create a
> couple of logical partitions on a single server that I can move later
> to another server to free disk space. Propably using lounge for that.
> 
> What I'm wondering though is how to move the data to a new node in
> that case. E.g. lets say I have node A with 16 logical partitions. Now
> I like to move 8 of them to a new node B. How can I move the data for
> only those 8 partitions to the new node? I can't just reimport the
> whole data as that would take too much time. Is that even possible to
> split an existing node that way?
> 
> Cheers,
> Stefan

Re: Moving Partitions

[Randall Leeds <ra...@gmail.com>]

On Mon, May 2, 2011 at 06:43, Stefan Podkowinski <sp...@gmail.com> wrote:
> Hello Couch Users
>
> As I'm about to develop a deployment strategy for my current company,
> theres one aspect left that keeps me puzzled. While couch seems to be
> pretty disk hungry I need a way to scale in terms of disk space and
> decided to setup oversharding right from the start. I plan to create a
> couple of logical partitions on a single server that I can move later
> to another server to free disk space. Propably using lounge for that.
>

The way Lounge works (and BigCouch also, because these days they use
the same scheme, last I checked) is to proxy the database name for the
logical pieces of your data.
That is, instead of having one database 'stuff' you have databases
called shards/00000000-1cffffff/stuff, each with their own hex range
representation of the data contained therein. A request to 'stuff'
composes the results from the calls to the various
shards/<hex-range>/stuff databases.

> What I'm wondering though is how to move the data to a new node in
> that case. E.g. lets say I have node A with 16 logical partitions. Now
> I like to move 8 of them to a new node B. How can I move the data for
> only those 8 partitions to the new node? I can't just reimport the
> whole data as that would take too much time. Is that even possible to
> split an existing node that way?
>

With this setup, moving a shard is a straightforward, normal replication.
In theory you could also perform filtered replication to split a shard
into two pieces with two target databases.
I haven't had to do this in practice yet with Lounge, but Cloudant may
have done this in production with BigCouch.
I don't know whether BigCouch automates any of these tasks. Lounge
currently does not, though I have a few scripts that help.