You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Stephen Bartell <sn...@gmail.com> on 2013/05/29 03:23:21 UTC
virtualizing databases
Hi all. I'm not sure if the subject really is the right terminology for what I'm trying to do.
What I want to do is maintain a routing table of database:destination pairs, so that on my couchdb I can name my databases however I want. I want to virtualize my database names.
I can see how this concept might be the basis for sharding. So maybe something like this already exists. But sharding is not my goal here. I want to use this mostly for maintenance. Oftentimes I need to do some work on a database, like trimming it down or filtering some stuff out. This work can take hours and I want the database always available to the end users. So for example, I have a database endpoint called "database" which maps to "database_a". In the background I might begin a filtered replication from "database_a" to "database_b". Users can still access "database" while "database_b" is being created. When the replication to "database_b" is done, I want to change my proxy table so that requests to "database" now go to "database_b". I can now forget about "database_a".
I'm sure many of you guys have run into this kind of thing before. What would you guys recommend.
Thanks!
Re: virtualizing databases
Posted by Adam Kocoloski <ko...@apache.org>.
I discussed COUCHDB-1736 on dev@ the other day and just copied my comment into JIRA. As you observed, an alias system is somewhat akin to a degenerate case of sharding. In BigCouch we store the shard map for each clustered database in a special system database that is local to each node in the cluster and is replicated between them.
The redirection to a remote CouchDB is interesting. I'd assumed something like that would be best accomplished via a reverse proxy in front of both servers.
Adam
On May 31, 2013, at 3:53 PM, Stephen Bartell <sn...@gmail.com> wrote:
> Thanks guys.
>
> Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
>
> Simon, my database list can get into the hundreds. Like you say, I think I'll need to use a proxy. I want this thing to bolt right into couchdb using daemon services or whatever other plugin approach is available. Once its activated, then it should start routing. The big question, kind of like what Benoit mentioned with aliases, is where does the configuration live? If a cluster of Couch's is involved, I think these configs should also replicate around. Do you know of anything like this that already exists?
>
> By the way, I gave https_global_handlers a try. This seems to work fine if the database is local. However, it doesn't work if the destination is remote and authentication is involved. Is this because the remote database does not have a session? Is it because auth isn't passed through the proxy? Is this a bug?
>
> Thanks for your thoughts!
>
> On May 28, 2013, at 8:21 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>
>> On May 29, 2013 3:23 AM, "Stephen Bartell" <sn...@gmail.com> wrote:
>>>
>>> Hi all. I'm not sure if the subject really is the right terminology for
>> what I'm trying to do.
>>>
>>> What I want to do is maintain a routing table of database:destination
>> pairs, so that on my couchdb I can name my databases however I want. I
>> want to virtualize my database names.
>>>
>>> I can see how this concept might be the basis for sharding. So maybe
>> something like this already exists. But sharding is not my goal here. I
>> want to use this mostly for maintenance. Oftentimes I need to do some work
>> on a database, like trimming it down or filtering some stuff out. This
>> work can take hours and I want the database always available to the end
>> users. So for example, I have a database endpoint called "database" which
>> maps to "database_a". In the background I might begin a filtered
>> replication from "database_a" to "database_b". Users can still access
>> "database" while "database_b" is being created. When the replication to
>> "database_b" is done, I want to change my proxy table so that requests to
>> "database" now go to "database_b". I can now forget about "database_a".
>>>
>>> I'm sure many of you guys have run into this kind of thing before. What
>> would you guys recommend.
>>>
>>> Thanks!
>>
>> DB aliases may sove that. There is a ticket about it. The questiin now is
>> where to store this meta info.
>>
>> - benot
>
Re: virtualizing databases
Posted by Stephen Bartell <sn...@gmail.com>.
No worries! I did not try with basic auth. My user facing apps all authenticate with cookies. And that is what does not work. I tested this by setting up a proxy handler to a remote database with the same security as the local database. I then accessed the database via futon and attempted to log in. It didn't work. Maybe proxying cookies is a bug then?
On May 31, 2013, at 1:43 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, May 31, 2013 at 10:41 PM, Stephen Bartell <sn...@gmail.com> wrote:
>> Benoit, I mentioned that in my email. I'll repaste it here:
>>
>> By the way, I gave https_global_handlers a try. This seems to work fine if the database is local. However, it doesn't work if the destination is remote and authentication is involved. Is this because the remote database does not have a session? Is it because auth isn't passed through the proxy? Is this a bug?
>>
>
> i misread that part sorry :) Did you try to pass the auth in the url ?
> ie. http://user:pass@....
>
> if it doesn't work, adding that feature should be easy.
>
> - benoit
>
>
>>
>> On May 31, 2013, at 1:34 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>
>>> On Fri, May 31, 2013 at 9:53 PM, Stephen Bartell <sn...@gmail.com> wrote:
>>>> Thanks guys.
>>>>
>>>> Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
>>>
>>> You can probably use the proxy handler in couch to do that
>>>
>>> [httpd_global_handlers]
>>> remotedb = {couch_httpd_proxy, handle_proxy_req,
>>> <<"http:/host:port/remotedb">>}
>>>
>>> - benoit
>>
Re: virtualizing databases
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, May 31, 2013 at 10:41 PM, Stephen Bartell <sn...@gmail.com> wrote:
> Benoit, I mentioned that in my email. I'll repaste it here:
>
> By the way, I gave https_global_handlers a try. This seems to work fine if the database is local. However, it doesn't work if the destination is remote and authentication is involved. Is this because the remote database does not have a session? Is it because auth isn't passed through the proxy? Is this a bug?
>
i misread that part sorry :) Did you try to pass the auth in the url ?
ie. http://user:pass@....
if it doesn't work, adding that feature should be easy.
- benoit
>
> On May 31, 2013, at 1:34 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>
>> On Fri, May 31, 2013 at 9:53 PM, Stephen Bartell <sn...@gmail.com> wrote:
>>> Thanks guys.
>>>
>>> Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
>>
>> You can probably use the proxy handler in couch to do that
>>
>> [httpd_global_handlers]
>> remotedb = {couch_httpd_proxy, handle_proxy_req,
>> <<"http:/host:port/remotedb">>}
>>
>> - benoit
>
Re: virtualizing databases
Posted by Stephen Bartell <sn...@gmail.com>.
Benoit, I mentioned that in my email. I'll repaste it here:
By the way, I gave https_global_handlers a try. This seems to work fine if the database is local. However, it doesn't work if the destination is remote and authentication is involved. Is this because the remote database does not have a session? Is it because auth isn't passed through the proxy? Is this a bug?
On May 31, 2013, at 1:34 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, May 31, 2013 at 9:53 PM, Stephen Bartell <sn...@gmail.com> wrote:
>> Thanks guys.
>>
>> Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
>
> You can probably use the proxy handler in couch to do that
>
> [httpd_global_handlers]
> remotedb = {couch_httpd_proxy, handle_proxy_req,
> <<"http:/host:port/remotedb">>}
>
> - benoit
Re: virtualizing databases
Posted by Stephen Bartell <sn...@gmail.com>.
I originally stated that I want to do this redirection at the couch level. Adam and Simon both mentioned that this is probably best done outside of couch. I think I've seen the light. Often times I still need access to the original database. If I am proxying at the couch level, then I can never get to that original database. If I needed to access a redirected database, then I would need to go through the proxy. But if I need to access the original database for whatever reason, I could still log into the couch server and access it.
Aliasing, remote or local, seems to be useful only when one version of that database is exposed at a time.
On May 31, 2013, at 1:34 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, May 31, 2013 at 9:53 PM, Stephen Bartell <sn...@gmail.com> wrote:
>> Thanks guys.
>>
>> Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
>
> You can probably use the proxy handler in couch to do that
>
> [httpd_global_handlers]
> remotedb = {couch_httpd_proxy, handle_proxy_req,
> <<"http:/host:port/remotedb">>}
>
> - benoit
Re: virtualizing databases
Posted by Benoit Chesneau <bc...@gmail.com>.
On Fri, May 31, 2013 at 9:53 PM, Stephen Bartell <sn...@gmail.com> wrote:
> Thanks guys.
>
> Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
You can probably use the proxy handler in couch to do that
[httpd_global_handlers]
remotedb = {couch_httpd_proxy, handle_proxy_req,
<<"http:/host:port/remotedb">>}
- benoit
Re: virtualizing databases
Posted by Stephen Bartell <sn...@gmail.com>.
Thanks guys.
Benoit, it looks like the alias improvement you are talking about is couchdb-1736? It still looks like its baking. What I need is something a little bigger than aliases though. I'd like the ability to redirect to a remote Couch.
Simon, my database list can get into the hundreds. Like you say, I think I'll need to use a proxy. I want this thing to bolt right into couchdb using daemon services or whatever other plugin approach is available. Once its activated, then it should start routing. The big question, kind of like what Benoit mentioned with aliases, is where does the configuration live? If a cluster of Couch's is involved, I think these configs should also replicate around. Do you know of anything like this that already exists?
By the way, I gave https_global_handlers a try. This seems to work fine if the database is local. However, it doesn't work if the destination is remote and authentication is involved. Is this because the remote database does not have a session? Is it because auth isn't passed through the proxy? Is this a bug?
Thanks for your thoughts!
On May 28, 2013, at 8:21 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On May 29, 2013 3:23 AM, "Stephen Bartell" <sn...@gmail.com> wrote:
>>
>> Hi all. I'm not sure if the subject really is the right terminology for
> what I'm trying to do.
>>
>> What I want to do is maintain a routing table of database:destination
> pairs, so that on my couchdb I can name my databases however I want. I
> want to virtualize my database names.
>>
>> I can see how this concept might be the basis for sharding. So maybe
> something like this already exists. But sharding is not my goal here. I
> want to use this mostly for maintenance. Oftentimes I need to do some work
> on a database, like trimming it down or filtering some stuff out. This
> work can take hours and I want the database always available to the end
> users. So for example, I have a database endpoint called "database" which
> maps to "database_a". In the background I might begin a filtered
> replication from "database_a" to "database_b". Users can still access
> "database" while "database_b" is being created. When the replication to
> "database_b" is done, I want to change my proxy table so that requests to
> "database" now go to "database_b". I can now forget about "database_a".
>>
>> I'm sure many of you guys have run into this kind of thing before. What
> would you guys recommend.
>>
>> Thanks!
>
> DB aliases may sove that. There is a ticket about it. The questiin now is
> where to store this meta info.
>
> - benot
Re: virtualizing databases
Posted by Benoit Chesneau <bc...@gmail.com>.
On May 29, 2013 3:23 AM, "Stephen Bartell" <sn...@gmail.com> wrote:
>
> Hi all. I'm not sure if the subject really is the right terminology for
what I'm trying to do.
>
> What I want to do is maintain a routing table of database:destination
pairs, so that on my couchdb I can name my databases however I want. I
want to virtualize my database names.
>
> I can see how this concept might be the basis for sharding. So maybe
something like this already exists. But sharding is not my goal here. I
want to use this mostly for maintenance. Oftentimes I need to do some work
on a database, like trimming it down or filtering some stuff out. This
work can take hours and I want the database always available to the end
users. So for example, I have a database endpoint called "database" which
maps to "database_a". In the background I might begin a filtered
replication from "database_a" to "database_b". Users can still access
"database" while "database_b" is being created. When the replication to
"database_b" is done, I want to change my proxy table so that requests to
"database" now go to "database_b". I can now forget about "database_a".
>
> I'm sure many of you guys have run into this kind of thing before. What
would you guys recommend.
>
> Thanks!
DB aliases may sove that. There is a ticket about it. The questiin now is
where to store this meta info.
- benot
Re: virtualizing databases
Posted by Simon Metson <si...@cloudant.com>.
How long is the list of databases you want to alias? Could you do it all in a proxy in front of couch?
On Wednesday, 29 May 2013 at 02:23, Stephen Bartell wrote:
> Hi all. I'm not sure if the subject really is the right terminology for what I'm trying to do.
>
> What I want to do is maintain a routing table of database:destination pairs, so that on my couchdb I can name my databases however I want. I want to virtualize my database names.
>
> I can see how this concept might be the basis for sharding. So maybe something like this already exists. But sharding is not my goal here. I want to use this mostly for maintenance. Oftentimes I need to do some work on a database, like trimming it down or filtering some stuff out. This work can take hours and I want the database always available to the end users. So for example, I have a database endpoint called "database" which maps to "database_a". In the background I might begin a filtered replication from "database_a" to "database_b". Users can still access "database" while "database_b" is being created. When the replication to "database_b" is done, I want to change my proxy table so that requests to "database" now go to "database_b". I can now forget about "database_a".
>
> I'm sure many of you guys have run into this kind of thing before. What would you guys recommend.
>
> Thanks!