[CouchDB 1.6] Use symlinks for database files?

[Kevin Qiu <ke...@points.com>]

Hi all,

I have a question regarding using symbolic links for {db}.couch file and
.{db}_design folder. First off, this is my usecase: we use couchdb 1.6.1,
and our data size has been growing rapidly to the point we have to archive
old data. We chose to use the filtered replication approach. We setup a
replication task from db A to db A' with a filter function that weeds out
documents older than X months (we have a metadata field that records the
document create date).

We would also like to seamlessly having our application start using the
filtered database (A') without having to change our application db
configuration to point to A'. Here's the question, is it possible to use
symbolic links to achieve this? i.e., have a symbolic link L ( (both .couch
file and _design folder) to point to A initially, and after the filtered
replication is done, re-point L to A'.

From my own experiments, it looks like this is possible, not without
caveats however. E.g., when L already exists, it works when I unlink L and
then do `ln -s A' L`. Everything seems to be fine. However, if I do `ln -s
A' L_tmp && mv -Tf L_tmp L` (atomic switch), it appears that couchdb fails
to recognize L being a pointer to another couchdb file, and shows that the
db contains 0 documents.

Any input is appreciated!

-Kevin

Re: [CouchDB 1.6] Use symlinks for database files?

[Kevin Qiu <ke...@points.com>]

Thanks. I've replied inline.

On Mon, Nov 28, 2016 at 1:19 PM, Joel Wallis Jucá <jo...@gmail.com>
wrote:

> Just curious: when you say your app will use the db A', do you app uses the
> CouchDB "way" to access data (aka the HTTP interface)? If it does, what's
> the point on using symlinks in the case you've just described?
>
>
Yes, we're using HTTP api to access couchdb. The point here is that we
don't have to change the configuration for individual services to point to
A'.



> FYI: this step (moving valid documents to a new db and discard
> deleted/invalid ones) is common with devs using CouchDB, but take note that
> if you're on a distributed environment it might not be a good idea (you
> need to have a deleted document being propagated to have proper eventual
> consistence).
>
>
We're not using couchdb in a distributed setup. We run 2 db instances, one
active and the other standby with haproxy in front.



> 2016-11-28 15:04 GMT-03:00 Kevin Qiu <ke...@points.com>:
>
> > Hi all,
> >
> > I have a question regarding using symbolic links for {db}.couch file and
> > .{db}_design folder. First off, this is my usecase: we use couchdb 1.6.1,
> > and our data size has been growing rapidly to the point we have to
> archive
> > old data. We chose to use the filtered replication approach. We setup a
> > replication task from db A to db A' with a filter function that weeds out
> > documents older than X months (we have a metadata field that records the
> > document create date).
> >
> > We would also like to seamlessly having our application start using the
> > filtered database (A') without having to change our application db
> > configuration to point to A'. Here's the question, is it possible to use
> > symbolic links to achieve this? i.e., have a symbolic link L ( (both
> .couch
> > file and _design folder) to point to A initially, and after the filtered
> > replication is done, re-point L to A'.
> >
> > From my own experiments, it looks like this is possible, not without
> > caveats however. E.g., when L already exists, it works when I unlink L
> and
> > then do `ln -s A' L`. Everything seems to be fine. However, if I do `ln
> -s
> > A' L_tmp && mv -Tf L_tmp L` (atomic switch), it appears that couchdb
> fails
> > to recognize L being a pointer to another couchdb file, and shows that
> the
> > db contains 0 documents.
> >
> > Any input is appreciated!
> >
> > -Kevin
> >
>
>
>
> --
> Joel Wallis Jucá
> joelwallis.com
>

Re: [CouchDB 1.6] Use symlinks for database files?

[Joel Wallis Jucá <jo...@gmail.com>]

Just curious: when you say your app will use the db A', do you app uses the
CouchDB "way" to access data (aka the HTTP interface)? If it does, what's
the point on using symlinks in the case you've just described?

FYI: this step (moving valid documents to a new db and discard
deleted/invalid ones) is common with devs using CouchDB, but take note that
if you're on a distributed environment it might not be a good idea (you
need to have a deleted document being propagated to have proper eventual
consistence).

2016-11-28 15:04 GMT-03:00 Kevin Qiu <ke...@points.com>:

> Hi all,
>
> I have a question regarding using symbolic links for {db}.couch file and
> .{db}_design folder. First off, this is my usecase: we use couchdb 1.6.1,
> and our data size has been growing rapidly to the point we have to archive
> old data. We chose to use the filtered replication approach. We setup a
> replication task from db A to db A' with a filter function that weeds out
> documents older than X months (we have a metadata field that records the
> document create date).
>
> We would also like to seamlessly having our application start using the
> filtered database (A') without having to change our application db
> configuration to point to A'. Here's the question, is it possible to use
> symbolic links to achieve this? i.e., have a symbolic link L ( (both .couch
> file and _design folder) to point to A initially, and after the filtered
> replication is done, re-point L to A'.
>
> From my own experiments, it looks like this is possible, not without
> caveats however. E.g., when L already exists, it works when I unlink L and
> then do `ln -s A' L`. Everything seems to be fine. However, if I do `ln -s
> A' L_tmp && mv -Tf L_tmp L` (atomic switch), it appears that couchdb fails
> to recognize L being a pointer to another couchdb file, and shows that the
> db contains 0 documents.
>
> Any input is appreciated!
>
> -Kevin
>



-- 
Joel Wallis Jucá
joelwallis.com