You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by CDR <cd...@free.fr> on 2012/06/20 17:59:16 UTC
database dynamic hot plug (attach / detach)
I would like to know if such behavior is possible :
For the runnning instance of couchdb server, ask to attach a db (from
another directory, even a usb key). Then detach it.
Also can a view gather data from more than one database.
Elsewhere, the solution is to start a new server on another listening
port, and then query the two servers. Then merge the view result, that
is getting complicated.
Regards
Re: database dynamic hot plug (attach / detach)
Posted by Benoit Chesneau <bc...@gmail.com>.
On Thu, Jun 21, 2012 at 1:47 PM, CDR <cd...@free.fr> wrote:
> Thank you to have enlighten my questions.
>
> To summarise, two choices:
> - start a server for each db, then merge queries from the n server to
> produce an output.
> - patch couchdb to add support of 'attach' and 'detach'
> functionalities. But still end with mutliple view (one per db per fs)
> that require result merging to produce an output.
>
> I think the first one sound more reasonable :)
> Just by curiosity, the attach and detach have to be coded in Erlang,
> involving file open and close. Is their a request queue, current IO or
> something like that inside couchdb? I guess I just need to watch server
> shutdown code.
>
> Regards.
>
There is an LRU cache keeping track of live database and check if
there are still alive. Imo it will be easier if we make the couch_core
embeddable and pluggable so we could have smth like :
couch_db:open(<<"dbname">>, [{dir, "/path/to/dbdir"}]) . It needs to
be a dir to handle compaction and other indexing features from couch.
The only question is how do you want to handle it using the HTTP api.
- benoit
Re: database dynamic hot plug (attach / detach)
Posted by CDR <cd...@free.fr>.
Thank you to have enlighten my questions.
To summarise, two choices:
- start a server for each db, then merge queries from the n server to
produce an output.
- patch couchdb to add support of 'attach' and 'detach'
functionalities. But still end with mutliple view (one per db per fs)
that require result merging to produce an output.
I think the first one sound more reasonable :)
Just by curiosity, the attach and detach have to be coded in Erlang,
involving file open and close. Is their a request queue, current IO or
something like that inside couchdb? I guess I just need to watch server
shutdown code.
Regards.
> An enhancement to make it pluggable does sound intriguing, though I'll
> note that when Ubuntu had a similar problem (distinct databases for
> distinct local users) they elected to run a separate couchdb process
> for each user.
>
> Bottom line: To get the feature you want, you'll need to patch
> CouchDB. Done well, I think there's a good chance we'd merge it into a
> future release.
Re: database dynamic hot plug (attach / detach)
Posted by Robert Newson <rn...@apache.org>.
CouchDB doesn't have a cache unless you mean that it has an open file descriptor on that file (which, obviously, would not be affected by the removal of a hard link). All the effects you've noticed seem to follow naturally from POSIX filesystem semantics.
CouchDB is a database server and requires all of its databases to be in the database_dir directory. It is not currently hot-pluggable. As you have also noticed, there are restrictions on using a single file system too.
An enhancement to make it pluggable does sound intriguing, though I'll note that when Ubuntu had a similar problem (distinct databases for distinct local users) they elected to run a separate couchdb process for each user.
Bottom line: To get the feature you want, you'll need to patch CouchDB. Done well, I think there's a good chance we'd merge it into a future release.
B.
On 21 Jun 2012, at 10:16, CDR wrote:
> I was sceptic, but I tried the link tricks (without restarting
> couchdb).
> - with a symbolic link, (softlink, ln -s)
> * CouchDB does not see the database (not a file).
> - with hard link (logical, ln- L)
> * CouchDB effectively see the new db through the link.
> * removing original file, couchdb still have the db file.
> * removing the hard link, couchdb should not have it anymore, FALSE!!
> ==>> detaching is not properly done because couchdb have a cache.
> /!\ HUGH LIMITATION, file and link have to be on the same partition.
> It is thus not possible to use this approach for removable filesystem.
>
> Couchdb have a cache, so we can access a db that is offline.
> Their is a way to see if the db file exist, _all_dbs list only dbs wiche
> are online (the db file is reachable).
>
> /!\ HAVE TO CHECK CURRENT (PENDING) REQUEST before detaching.
>
> Alternate, but database statistic on access required, is to lock the DB.
> The easiest way is to lock de db rejecting all update with
> "validate_doc_update" and to wait at least 1 seconde before
> disconnecting the external db file (explanantion: couch db flush every
> second as in second part in "Batch Mode" paragraphe at
> http://guide.couchdb.org/draft/performance.html ).
> So even if batch mode enabled, no data losses (if small data).
>
> REQUIERED DB ACCESS STATISTICS, because of following case :
> However I have tried a big attachement, requiering more than 1 second of
> file IO (100Mo). This does not result in a direct writting, but a long
> caching in memory (compressing?) and only a long time (more than 1s,
> roughtly 30s) after, write is performed to the disk.
> This is not acceptable to detach the database.
>
> A way to monitor db activity would be nice, but so far, I have not found
> such in the couchDB documentation.
>
> Does it exist at least? internally I guesss so.
>
> Finally the db file is constantly opened (sudo lsof -r 1 dbfile). if the
> db can garantee their is no more access (all flushed) this is not really
> a trouble, but it would be proper to have a clean fcolse.
>
> WHY :
> - I work on a filesystem project, and I would require to mount unmount
> db like we do with filesystem. Their is a real need to cary the db fs,
> and not jsut files on standard fs (lacks of information).
>
> Regards
>
> Le mercredi 20 juin 2012 à 21:19 +0200, Dave Cottlehuber a écrit :
>> On 20 June 2012 17:59, CDR <cd...@free.fr> wrote:
>>> I would like to know if such behavior is possible :
>>> For the runnning instance of couchdb server, ask to attach a db (from
>>> another directory, even a usb key). Then detach it.
>>
>> You could use a soft link to achieve this. Just don't compact the db.
>> You need to think about where the views will go also.
>>
>> I'm assuming that you are not planning to do this on a "real"
>> server environment so the risk of having incorrect on-disk
>> formats (they change over releases) and screw-ups due to compacting
>> and view-building are less of a concern.
>>
>> If its a server I'd avoid the monkeying around. Why is this necessary?
>>
>>> Also can a view gather data from more than one database.
>>
>> No, I don't know if the other forks of CouchDB support this tho.
>>
>>> Elsewhere, the solution is to start a new server on another listening
>>> port, and then query the two servers. Then merge the view result, that
>>> is getting complicated.
>>
>> That's called a JOIN I believe.
>>
>> A+
>> Dave
>
>
Re: database dynamic hot plug (attach / detach)
Posted by CDR <cd...@free.fr>.
I was sceptic, but I tried the link tricks (without restarting
couchdb).
- with a symbolic link, (softlink, ln -s)
* CouchDB does not see the database (not a file).
- with hard link (logical, ln- L)
* CouchDB effectively see the new db through the link.
* removing original file, couchdb still have the db file.
* removing the hard link, couchdb should not have it anymore, FALSE!!
==>> detaching is not properly done because couchdb have a cache.
/!\ HUGH LIMITATION, file and link have to be on the same partition.
It is thus not possible to use this approach for removable filesystem.
Couchdb have a cache, so we can access a db that is offline.
Their is a way to see if the db file exist, _all_dbs list only dbs wiche
are online (the db file is reachable).
/!\ HAVE TO CHECK CURRENT (PENDING) REQUEST before detaching.
Alternate, but database statistic on access required, is to lock the DB.
The easiest way is to lock de db rejecting all update with
"validate_doc_update" and to wait at least 1 seconde before
disconnecting the external db file (explanantion: couch db flush every
second as in second part in "Batch Mode" paragraphe at
http://guide.couchdb.org/draft/performance.html ).
So even if batch mode enabled, no data losses (if small data).
REQUIERED DB ACCESS STATISTICS, because of following case :
However I have tried a big attachement, requiering more than 1 second of
file IO (100Mo). This does not result in a direct writting, but a long
caching in memory (compressing?) and only a long time (more than 1s,
roughtly 30s) after, write is performed to the disk.
This is not acceptable to detach the database.
A way to monitor db activity would be nice, but so far, I have not found
such in the couchDB documentation.
Does it exist at least? internally I guesss so.
Finally the db file is constantly opened (sudo lsof -r 1 dbfile). if the
db can garantee their is no more access (all flushed) this is not really
a trouble, but it would be proper to have a clean fcolse.
WHY :
- I work on a filesystem project, and I would require to mount unmount
db like we do with filesystem. Their is a real need to cary the db fs,
and not jsut files on standard fs (lacks of information).
Regards
Le mercredi 20 juin 2012 à 21:19 +0200, Dave Cottlehuber a écrit :
> On 20 June 2012 17:59, CDR <cd...@free.fr> wrote:
> > I would like to know if such behavior is possible :
> > For the runnning instance of couchdb server, ask to attach a db (from
> > another directory, even a usb key). Then detach it.
>
> You could use a soft link to achieve this. Just don't compact the db.
> You need to think about where the views will go also.
>
> I'm assuming that you are not planning to do this on a "real"
> server environment so the risk of having incorrect on-disk
> formats (they change over releases) and screw-ups due to compacting
> and view-building are less of a concern.
>
> If its a server I'd avoid the monkeying around. Why is this necessary?
>
> > Also can a view gather data from more than one database.
>
> No, I don't know if the other forks of CouchDB support this tho.
>
> > Elsewhere, the solution is to start a new server on another listening
> > port, and then query the two servers. Then merge the view result, that
> > is getting complicated.
>
> That's called a JOIN I believe.
>
> A+
> Dave
Re: database dynamic hot plug (attach / detach)
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 20 June 2012 17:59, CDR <cd...@free.fr> wrote:
> I would like to know if such behavior is possible :
> For the runnning instance of couchdb server, ask to attach a db (from
> another directory, even a usb key). Then detach it.
You could use a soft link to achieve this. Just don't compact the db.
You need to think about where the views will go also.
I'm assuming that you are not planning to do this on a "real"
server environment so the risk of having incorrect on-disk
formats (they change over releases) and screw-ups due to compacting
and view-building are less of a concern.
If its a server I'd avoid the monkeying around. Why is this necessary?
> Also can a view gather data from more than one database.
No, I don't know if the other forks of CouchDB support this tho.
> Elsewhere, the solution is to start a new server on another listening
> port, and then query the two servers. Then merge the view result, that
> is getting complicated.
That's called a JOIN I believe.
A+
Dave