You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Rick Jarvis <ri...@magicmail.mooo.com> on 2021/07/04 17:02:22 UTC
Per subdomain DBs with Node
Hey all
I have an app which serves multiple orgs with individual databases, by way of subdomain, ORGNAME.myapp.foo . My current method of attaching the database to the ExpressJS requests via middleware is like this:
app.use(function(req, res, next) {
const orgname = req.headers.host.split('.')[0].toLowerCase();
req.couch = nano.db.use(orgname);
next();
}
I often wonder if this is the best way of doing it. Is there a better way?
I could for instance, attach it just once, to the session (stored in Redis):
app.use(function(req, res, next) {
if(!req.session.couch) {
const orgname = req.headers.host.split('.')[0].toLowerCase();
req.session.couch = nano.db.use(orgname);
}
next();
}
But I guess that depends on size of the Couch instance (is it just metadata, or something more tangible?), and other aspects which I perhaps haven’t considered.
I’d love to hear some thoughts on this?
Thank you!
R
Re: Per subdomain DBs with Node
Posted by Jan Lehnardt <ja...@apache.org>.
Hi Rick,
your original implementation strikes me as better than the other one.
Since storing something in Redis means serialising it, you won’t get the benefit of any sockets being kept open, and I’m not even sure you can make that work properly.
If you are worried about having a db instance open per request vs once-per-server-per-user, how about this slight modification to your original pattern. This is something we’re using in Opservatory* for the same reason:
let dbs = {}
function getDb(orgname) {
if (!dbs[orgname] {
// cache db instance if not already cached
dbs[orgname] = nano.db.use(orgname);
}
// return cached db instance
return dbs[orgname]
}
app.use(function(req, res, next) {
const orgname = req.headers.host.split('.')[0].toLowerCase();
req.couch = getDb(orgname)
next();
}
If you have very long running servers and high user churn, you also want to make sure you can drop cached instances from `dbs` when a user gets deleted. Or you add a max length check to this and drop the least recently used instance, but you get the idea.
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
*24/7 Observation for your CouchDB Instances:
https://opservatory.app
> On 4. Jul 2021, at 19:02, Rick Jarvis <ri...@magicmail.mooo.com> wrote:
>
> Hey all
>
> I have an app which serves multiple orgs with individual databases, by way of subdomain, ORGNAME.myapp.foo . My current method of attaching the database to the ExpressJS requests via middleware is like this:
>
> app.use(function(req, res, next) {
> const orgname = req.headers.host.split('.')[0].toLowerCase();
> req.couch = nano.db.use(orgname);
> next();
> }
>
> I often wonder if this is the best way of doing it. Is there a better way?
>
> I could for instance, attach it just once, to the session (stored in Redis):
>
> app.use(function(req, res, next) {
> if(!req.session.couch) {
> const orgname = req.headers.host.split('.')[0].toLowerCase();
> req.session.couch = nano.db.use(orgname);
> }
> next();
> }
>
> But I guess that depends on size of the Couch instance (is it just metadata, or something more tangible?), and other aspects which I perhaps haven’t considered.
>
> I’d love to hear some thoughts on this?
>
> Thank you!
> R
Re: Per subdomain DBs with Node
Posted by Sinan Gabel <si...@gmail.com>.
Here's some further info:
https://github.com/apache/couchdb-nano#pool-size-and-open-sockets
https://nodejs.org/api/http.html#http_new_agent_options
However, I would not worry too much about the number of sockets at first.
Better to make some performance* tests that fits your use case: if the
tests work with the default settings then all is good.
*) I use this performance tester:
https://httpd.apache.org/docs/2.4/programs/ab.html
On Tue, 6 Jul 2021 at 12:11, Jan Lehnardt <ja...@apache.org> wrote:
> Hi Rick,
>
> > On 5. Jul 2021, at 18:14, Rick Jarvis <ri...@magicmail.mooo.com> wrote:
> >
> > That looks like a good addition. Couple of quick questions on that:
> >
> > Does that keep the socket open indefinitely whilst the Node process is
> running, and if so does it create much benefit?
> >
> > Is ’nano.db.use’ actually opening the socket to the DB then? And how
> many sockets is a reasonable expectation (I guess that’s how long is a
> piece of string though?!)?
>
>
> The exact socket handling I’m not sure about, you’d have to look at how
> nano is actually implemented. All this will also be affected by your
> keepalive settings, so the answer is probably something you’ll have to look
> for yourself, but you should have all pointers now :)
>
>
> > I guess I could make it ‘global.dbs’ and access the DBs easily through
> other modules etc then?
>
> Yeah that works too. In our case, all database instances are created in
> the same module that all other modules then use to get their instances from
> that one central module, so the “cache” can be module-global for us. Maybe
> that pattern also works for you. I hear people have strong opinions about
> global variables.
>
> Best
> Jan
> —
>
> >
> > Thanks!
> >
> >
> >
> > From: Jan Lehnardt <ja...@apache.org>
> > Reply: user@couchdb.apache.org <us...@couchdb.apache.org>
> > Date: 5 July 2021 at 10:34:29
> > To: user@couchdb.apache.org <us...@couchdb.apache.org>
> > Subject: Re: Per subdomain DBs with Node
> >
> > Hi Rick,
> >
> > your original implementation strikes me as better than the other one.
> >
> > Since storing something in Redis means serialising it, you won’t get the
> benefit of any sockets being kept open, and I’m not even sure you can make
> that work properly.
> >
> > If you are worried about having a db instance open per request vs
> once-per-server-per-user, how about this slight modification to your
> original pattern. This is something we’re using in Opservatory* for the
> same reason:
> >
> > let dbs = {}
> >
> > function getDb(orgname) {
> > if (!dbs[orgname] {
> > // cache db instance if not already cached
> > dbs[orgname] = nano.db.use(orgname);
> > }
> > // return cached db instance
> > return dbs[orgname]
> > }
> >
> > app.use(function(req, res, next) {
> > const orgname = req.headers.host.split('.')[0].toLowerCase();
> > req.couch = getDb(orgname)
> > next();
> > }
> >
> > If you have very long running servers and high user churn, you also want
> to make sure you can drop cached instances from `dbs` when a user gets
> deleted. Or you add a max length check to this and drop the least recently
> used instance, but you get the idea.
> >
> > Best
> > Jan
> > —
> >
> > Professional Support for Apache CouchDB:
> > https://neighbourhood.ie/couchdb-support/
> >
> > *24/7 Observation for your CouchDB Instances:
> > https://opservatory.app
> >
> >> On 4. Jul 2021, at 19:02, Rick Jarvis <ri...@magicmail.mooo.com>
> wrote:
> >>
> >> Hey all
> >>
> >> I have an app which serves multiple orgs with individual databases, by
> way of subdomain, ORGNAME.myapp.foo . My current method of attaching the
> database to the ExpressJS requests via middleware is like this:
> >>
> >> app.use(function(req, res, next) {
> >> const orgname = req.headers.host.split('.')[0].toLowerCase();
> >> req.couch = nano.db.use(orgname);
> >> next();
> >> }
> >>
> >> I often wonder if this is the best way of doing it. Is there a better
> way?
> >>
> >> I could for instance, attach it just once, to the session (stored in
> Redis):
> >>
> >> app.use(function(req, res, next) {
> >> if(!req.session.couch) {
> >> const orgname = req.headers.host.split('.')[0].toLowerCase();
> >> req.session.couch = nano.db.use(orgname);
> >> }
> >> next();
> >> }
> >>
> >> But I guess that depends on size of the Couch instance (is it just
> metadata, or something more tangible?), and other aspects which I perhaps
> haven’t considered.
> >>
> >> I’d love to hear some thoughts on this?
> >>
> >> Thank you!
> >> R
>
>
Re: Per subdomain DBs with Node
Posted by Jan Lehnardt <ja...@apache.org>.
Hi Rick,
> On 5. Jul 2021, at 18:14, Rick Jarvis <ri...@magicmail.mooo.com> wrote:
>
> That looks like a good addition. Couple of quick questions on that:
>
> Does that keep the socket open indefinitely whilst the Node process is running, and if so does it create much benefit?
>
> Is ’nano.db.use’ actually opening the socket to the DB then? And how many sockets is a reasonable expectation (I guess that’s how long is a piece of string though?!)?
The exact socket handling I’m not sure about, you’d have to look at how nano is actually implemented. All this will also be affected by your keepalive settings, so the answer is probably something you’ll have to look for yourself, but you should have all pointers now :)
> I guess I could make it ‘global.dbs’ and access the DBs easily through other modules etc then?
Yeah that works too. In our case, all database instances are created in the same module that all other modules then use to get their instances from that one central module, so the “cache” can be module-global for us. Maybe that pattern also works for you. I hear people have strong opinions about global variables.
Best
Jan
—
>
> Thanks!
>
>
>
> From: Jan Lehnardt <ja...@apache.org>
> Reply: user@couchdb.apache.org <us...@couchdb.apache.org>
> Date: 5 July 2021 at 10:34:29
> To: user@couchdb.apache.org <us...@couchdb.apache.org>
> Subject: Re: Per subdomain DBs with Node
>
> Hi Rick,
>
> your original implementation strikes me as better than the other one.
>
> Since storing something in Redis means serialising it, you won’t get the benefit of any sockets being kept open, and I’m not even sure you can make that work properly.
>
> If you are worried about having a db instance open per request vs once-per-server-per-user, how about this slight modification to your original pattern. This is something we’re using in Opservatory* for the same reason:
>
> let dbs = {}
>
> function getDb(orgname) {
> if (!dbs[orgname] {
> // cache db instance if not already cached
> dbs[orgname] = nano.db.use(orgname);
> }
> // return cached db instance
> return dbs[orgname]
> }
>
> app.use(function(req, res, next) {
> const orgname = req.headers.host.split('.')[0].toLowerCase();
> req.couch = getDb(orgname)
> next();
> }
>
> If you have very long running servers and high user churn, you also want to make sure you can drop cached instances from `dbs` when a user gets deleted. Or you add a max length check to this and drop the least recently used instance, but you get the idea.
>
> Best
> Jan
> —
>
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
> *24/7 Observation for your CouchDB Instances:
> https://opservatory.app
>
>> On 4. Jul 2021, at 19:02, Rick Jarvis <ri...@magicmail.mooo.com> wrote:
>>
>> Hey all
>>
>> I have an app which serves multiple orgs with individual databases, by way of subdomain, ORGNAME.myapp.foo . My current method of attaching the database to the ExpressJS requests via middleware is like this:
>>
>> app.use(function(req, res, next) {
>> const orgname = req.headers.host.split('.')[0].toLowerCase();
>> req.couch = nano.db.use(orgname);
>> next();
>> }
>>
>> I often wonder if this is the best way of doing it. Is there a better way?
>>
>> I could for instance, attach it just once, to the session (stored in Redis):
>>
>> app.use(function(req, res, next) {
>> if(!req.session.couch) {
>> const orgname = req.headers.host.split('.')[0].toLowerCase();
>> req.session.couch = nano.db.use(orgname);
>> }
>> next();
>> }
>>
>> But I guess that depends on size of the Couch instance (is it just metadata, or something more tangible?), and other aspects which I perhaps haven’t considered.
>>
>> I’d love to hear some thoughts on this?
>>
>> Thank you!
>> R
Re: Per subdomain DBs with Node
Posted by Rick Jarvis <ri...@magicmail.mooo.com>.
That looks like a good addition. Couple of quick questions on that:
Does that keep the socket open indefinitely whilst the Node process is running, and if so does it create much benefit?
Is ’nano.db.use’ actually opening the socket to the DB then? And how many sockets is a reasonable expectation (I guess that’s how long is a piece of string though?!)?
I guess I could make it ‘global.dbs’ and access the DBs easily through other modules etc then?
Thanks!
From: Jan Lehnardt <ja...@apache.org>
Reply: user@couchdb.apache.org <us...@couchdb.apache.org>
Date: 5 July 2021 at 10:34:29
To: user@couchdb.apache.org <us...@couchdb.apache.org>
Subject: Re: Per subdomain DBs with Node
Hi Rick,
your original implementation strikes me as better than the other one.
Since storing something in Redis means serialising it, you won’t get the benefit of any sockets being kept open, and I’m not even sure you can make that work properly.
If you are worried about having a db instance open per request vs once-per-server-per-user, how about this slight modification to your original pattern. This is something we’re using in Opservatory* for the same reason:
let dbs = {}
function getDb(orgname) {
if (!dbs[orgname] {
// cache db instance if not already cached
dbs[orgname] = nano.db.use(orgname);
}
// return cached db instance
return dbs[orgname]
}
app.use(function(req, res, next) {
const orgname = req.headers.host.split('.')[0].toLowerCase();
req.couch = getDb(orgname)
next();
}
If you have very long running servers and high user churn, you also want to make sure you can drop cached instances from `dbs` when a user gets deleted. Or you add a max length check to this and drop the least recently used instance, but you get the idea.
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
*24/7 Observation for your CouchDB Instances:
https://opservatory.app
> On 4. Jul 2021, at 19:02, Rick Jarvis <ri...@magicmail.mooo.com> wrote:
>
> Hey all
>
> I have an app which serves multiple orgs with individual databases, by way of subdomain, ORGNAME.myapp.foo . My current method of attaching the database to the ExpressJS requests via middleware is like this:
>
> app.use(function(req, res, next) {
> const orgname = req.headers.host.split('.')[0].toLowerCase();
> req.couch = nano.db.use(orgname);
> next();
> }
>
> I often wonder if this is the best way of doing it. Is there a better way?
>
> I could for instance, attach it just once, to the session (stored in Redis):
>
> app.use(function(req, res, next) {
> if(!req.session.couch) {
> const orgname = req.headers.host.split('.')[0].toLowerCase();
> req.session.couch = nano.db.use(orgname);
> }
> next();
> }
>
> But I guess that depends on size of the Couch instance (is it just metadata, or something more tangible?), and other aspects which I perhaps haven’t considered.
>
> I’d love to hear some thoughts on this?
>
> Thank you!
> R