You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by svilen <az...@svilendobrev.com> on 2012/10/07 07:59:33 UTC
can i listen to changes of whole server, not per-db?
g'day
i'm going to have thousands of databases. They are all of similar kind,
but belong to different (groups of) users, so can't be bundled into one.
And setupping 10000 connections to couchdb/dbXXXXX/_changes doesn't
seem very neat, even if on same machine.
any idea? even if it means recompiling with some re-config, or
hacking source somewhere, so be it - just point me where/what..
ciao
svil
Re: can i listen to changes of whole server, not per-db?
Posted by svilen <az...@svilendobrev.com>.
ah yes i also had the idea of summarizing all dbs into one just to
aggregate the _changes, and abandoned it.
For now i'm walking a set of databases and give them some
limited attention that handles changes. It very much looks like
web-framework routing and request-handling ;-) add some FSMs..
Was even thinking to setup it that way, and to have a dispatcher
kick proper urls according to database+message-type-and-state, i.e.
translating from stream of changes into stream of commands-as-urls
which would be plain http served..
svil
On Wed, 10 Oct 2012 10:42:27
+0100 Martin Higham <ma...@ocasta.co.uk> wrote:
> We had a similar problem. Initially we had each database replicate
> appropriate documents (via a replication filter) to a queue database
> and just listened for changes to that. The resource overhead for this
> when most databases are dormant didn't make make sense. So we now
> have a custom handler that invokes some python whenever a PUT, PUSH
> or DELETE occur to a subset of databases. I've put this up on GitHub -
> https://github.com/ocastalabs/couchdb_task_handler.
>
> This handler is dependant on CouchDB including the document ID in the
> response headers which it currently doesn't so we had to make a small
> change to the CouchDB source. You can find this at
> https://github.com/ocastalabs/couchdb/tree/id_and_rev_in_response_headers.
> We've raised a Pull Request for this (
> https://github.com/apache/couchdb/pull/32) and it would be great to
> get it into 1.3
>
> Martin
>
>
>
> On 7 October 2012 11:37, svilen <az...@svilendobrev.com> wrote:
>
> > no worries, not that in a hurry.
> > i'll put some workaround in the mean time..
> > still prototyping anyway.
> >
> > ciao
> > svil
> >
> > On Sun, 7 Oct 2012 12:18:29 +0200
> > Benoit Chesneau <bc...@gmail.com> wrote:
> >
> > > On Sun, Oct 7, 2012 at 10:25 AM, svilen <az...@svilendobrev.com>
> > > wrote:
> > > >> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> > > >> > g'day
> > > >> >
> > > >> > i'm going to have thousands of databases. They are all of
> > > >> > similar kind, but belong to different (groups of) users, so
> > > >> > can't be bundled into one. And setupping 10000 connections to
> > > >> > couchdb/dbXXXXX/_changes doesn't seem very neat, even if on
> > > >> > same machine.
> > > >
> > > > or maybe i can traverse all of them (i think i can figure out
> > > > the db names), paying them some limited attention each - e.g.
> > > > use short-poll with limit 10 changes.. poor-man's time-share
> > > > scheduler ;-)
> > > >
> > > > svil
> > >
> > > Well it will be costly, depending on the number of databases you
> > > have. If you need it fast you can use rcouch waiting that I port
> > > the code to couch. Can be done next week.
> > >
> > > - benoît
> >
Re: can i listen to changes of whole server, not per-db?
Posted by Martin Higham <ma...@ocasta.co.uk>.
We had a similar problem. Initially we had each database replicate
appropriate documents (via a replication filter) to a queue database and
just listened for changes to that. The resource overhead for this when most
databases are dormant didn't make make sense. So we now have a custom
handler that invokes some python whenever a PUT, PUSH or DELETE occur to a
subset of databases. I've put this up on GitHub -
https://github.com/ocastalabs/couchdb_task_handler.
This handler is dependant on CouchDB including the document ID in the
response headers which it currently doesn't so we had to make a small
change to the CouchDB source. You can find this at
https://github.com/ocastalabs/couchdb/tree/id_and_rev_in_response_headers.
We've raised a Pull Request for this (
https://github.com/apache/couchdb/pull/32) and it would be great to get it
into 1.3
Martin
On 7 October 2012 11:37, svilen <az...@svilendobrev.com> wrote:
> no worries, not that in a hurry.
> i'll put some workaround in the mean time..
> still prototyping anyway.
>
> ciao
> svil
>
> On Sun, 7 Oct 2012 12:18:29 +0200
> Benoit Chesneau <bc...@gmail.com> wrote:
>
> > On Sun, Oct 7, 2012 at 10:25 AM, svilen <az...@svilendobrev.com> wrote:
> > >> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> > >> > g'day
> > >> >
> > >> > i'm going to have thousands of databases. They are all of similar
> > >> > kind, but belong to different (groups of) users, so can't be
> > >> > bundled into one. And setupping 10000 connections to
> > >> > couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
> > >> > machine.
> > >
> > > or maybe i can traverse all of them (i think i can figure out the
> > > db names), paying them some limited attention each - e.g. use
> > > short-poll with limit 10 changes.. poor-man's time-share
> > > scheduler ;-)
> > >
> > > svil
> >
> > Well it will be costly, depending on the number of databases you have.
> > If you need it fast you can use rcouch waiting that I port the code to
> > couch. Can be done next week.
> >
> > - benoît
>
Re: can i listen to changes of whole server, not per-db?
Posted by svilen <az...@svilendobrev.com>.
no worries, not that in a hurry.
i'll put some workaround in the mean time..
still prototyping anyway.
ciao
svil
On Sun, 7 Oct 2012 12:18:29 +0200
Benoit Chesneau <bc...@gmail.com> wrote:
> On Sun, Oct 7, 2012 at 10:25 AM, svilen <az...@svilendobrev.com> wrote:
> >> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> >> > g'day
> >> >
> >> > i'm going to have thousands of databases. They are all of similar
> >> > kind, but belong to different (groups of) users, so can't be
> >> > bundled into one. And setupping 10000 connections to
> >> > couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
> >> > machine.
> >
> > or maybe i can traverse all of them (i think i can figure out the
> > db names), paying them some limited attention each - e.g. use
> > short-poll with limit 10 changes.. poor-man's time-share
> > scheduler ;-)
> >
> > svil
>
> Well it will be costly, depending on the number of databases you have.
> If you need it fast you can use rcouch waiting that I port the code to
> couch. Can be done next week.
>
> - benoît
Re: can i listen to changes of whole server, not per-db?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Oct 7, 2012 at 10:25 AM, svilen <az...@svilendobrev.com> wrote:
>> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
>> > g'day
>> >
>> > i'm going to have thousands of databases. They are all of similar
>> > kind, but belong to different (groups of) users, so can't be
>> > bundled into one. And setupping 10000 connections to
>> > couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
>> > machine.
>
> or maybe i can traverse all of them (i think i can figure out the
> db names), paying them some limited attention each - e.g. use short-poll
> with limit 10 changes.. poor-man's time-share scheduler ;-)
>
> svil
Well it will be costly, depending on the number of databases you have.
If you need it fast you can use rcouch waiting that I port the code to
couch. Can be done next week.
- benoît
Re: can i listen to changes of whole server, not per-db?
Posted by svilen <az...@svilendobrev.com>.
> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> > g'day
> >
> > i'm going to have thousands of databases. They are all of similar
> > kind, but belong to different (groups of) users, so can't be
> > bundled into one. And setupping 10000 connections to
> > couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
> > machine.
or maybe i can traverse all of them (i think i can figure out the
db names), paying them some limited attention each - e.g. use short-poll
with limit 10 changes.. poor-man's time-share scheduler ;-)
svil
Re: can i listen to changes of whole server, not per-db?
Posted by svilen <az...@svilendobrev.com>.
yeah, +1 for this one.
it would be rather useful for admins too, to know whether there's
activity on whole store.
svil
On Sun, 7 Oct 2012 09:43:15 +0200
Benoit Chesneau <bc...@gmail.com> wrote:
> On Sun, Oct 7, 2012 at 9:27 AM, Dave Cottlehuber <dc...@jsonified.com>
> wrote:
> > On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> >> g'day
> >>
> >> i'm going to have thousands of databases. They are all of similar
> >> kind, but belong to different (groups of) users, so can't be
> >> bundled into one. And setupping 10000 connections to
> >> couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
> >> machine.
> >
> > Not possible today unfortunately.
> >
> > Has anybody got any numbers or experience on practical limits of
> > _changes feeds connections to a single server?
> >
> > A+
> > Dave
>
>
> For rcouch I have written a plugin that listen on all dbs changes:
>
> https://github.com/refuge/couch_dbupdates
>
> It only notify you when a db is updated for any reason. Then you can
> use _changes to get latest changes for this db.
>
>
> I'm actually actively working on porting patches to couch and prepare
> some branches. If some think it would be interesting to have it in
> couch I can port it as well.
>
> - benoît
Re: can i listen to changes of whole server, not per-db?
Posted by svilen <az...@svilendobrev.com>.
how about hierarchical replication then ?
kind-a
- have a list of observed dbs
- listen to some special db/feed that gets updated when some
of above dbs changes
- replication over above scheme (i.e. replicate continiously the
speical db and replicate others once-off at each update)
i may do the above, as otherwise there will be 10000s of connections
doing nothing (but still kept alive).
svil
On Sun, 7 Oct 2012 09:43:15 +0200
Benoit Chesneau <bc...@gmail.com> wrote:
> On Sun, Oct 7, 2012 at 9:27 AM, Dave Cottlehuber <dc...@jsonified.com>
> wrote:
> > On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> >> g'day
> >>
> >> i'm going to have thousands of databases. They are all of similar
> >> kind, but belong to different (groups of) users, so can't be
> >> bundled into one. And setupping 10000 connections to
> >> couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
> >> machine.
> >
> > Not possible today unfortunately.
> >
> > Has anybody got any numbers or experience on practical limits of
> > _changes feeds connections to a single server?
> >
> > A+
> > Dave
>
>
> For rcouch I have written a plugin that listen on all dbs changes:
>
> https://github.com/refuge/couch_dbupdates
>
> It only notify you when a db is updated for any reason. Then you can
> use _changes to get latest changes for this db.
>
>
> I'm actually actively working on porting patches to couch and prepare
> some branches. If some think it would be interesting to have it in
> couch I can port it as well.
>
> - benoît
Re: can i listen to changes of whole server, not per-db?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Sun, Oct 7, 2012 at 9:27 AM, Dave Cottlehuber <dc...@jsonified.com> wrote:
> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
>> g'day
>>
>> i'm going to have thousands of databases. They are all of similar kind,
>> but belong to different (groups of) users, so can't be bundled into one.
>> And setupping 10000 connections to couchdb/dbXXXXX/_changes doesn't
>> seem very neat, even if on same machine.
>
> Not possible today unfortunately.
>
> Has anybody got any numbers or experience on practical limits of
> _changes feeds connections to a single server?
>
> A+
> Dave
For rcouch I have written a plugin that listen on all dbs changes:
https://github.com/refuge/couch_dbupdates
It only notify you when a db is updated for any reason. Then you can
use _changes to get latest changes for this db.
I'm actually actively working on porting patches to couch and prepare
some branches. If some think it would be interesting to have it in
couch I can port it as well.
- benoît
Re: can i listen to changes of whole server, not per-db?
Posted by svilen <az...@svilendobrev.com>.
what i am trying to do is similar to syncpoint. effectively couchdb
is the api-message-carrier. The idea of database-as-security-domain
also fits perfectly - users would see/replicate only dbs that they
belong to. But - N users working in M databases means listening to N+M
changes-feeds and post-processing them.
lumping all in one db, and filtering that into
internally-filtered partial replications (a filter per user) seems ...
rather tricky. can a client-replicator talk to http-proxy that looks
like normal couchdb but is (invisibly-to-client) routed into a
filtered-replication?
ah i should think how these can scale too..
svil
On Sun, 7 Oct 2012 09:27:18 +0200
Dave Cottlehuber <dc...@jsonified.com> wrote:
> On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> > g'day
> >
> > i'm going to have thousands of databases. They are all of similar
> > kind, but belong to different (groups of) users, so can't be
> > bundled into one. And setupping 10000 connections to
> > couchdb/dbXXXXX/_changes doesn't seem very neat, even if on same
> > machine.
>
> Not possible today unfortunately.
>
> Has anybody got any numbers or experience on practical limits of
> _changes feeds connections to a single server?
>
> A+
> Dave
Re: can i listen to changes of whole server, not per-db?
Posted by Dave Cottlehuber <dc...@jsonified.com>.
On 7 October 2012 07:59, svilen <az...@svilendobrev.com> wrote:
> g'day
>
> i'm going to have thousands of databases. They are all of similar kind,
> but belong to different (groups of) users, so can't be bundled into one.
> And setupping 10000 connections to couchdb/dbXXXXX/_changes doesn't
> seem very neat, even if on same machine.
Not possible today unfortunately.
Has anybody got any numbers or experience on practical limits of
_changes feeds connections to a single server?
A+
Dave