You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Antony Blakey <an...@gmail.com> on 2008/11/05 00:47:50 UTC
Architectural advice requested, and some 1.0 suggestions.
Hi, [ I'm posting to both -user and -dev because IMO it's relevant to
both ]
I'm in the middle of a trial port of a system from postgresql to
couchdb. The system has a distribution/replication requirement due
primarily to unreliable connectivity and limited end-to-end bandwidth
(and high latency) over a geographically dispersed set of users.
I have two requirements that don't fit well with the basic couchdb
model. I've developed solutions for which I'd appreciate some
feedback. Also it's a request for the notification mechanism to be
improved, something like _external to moved to the core, and/or an in-
process erlang plugin model that could replace notification/_external-
based techniques.
My solution is a generalization of the GeoCouch technique: http://vmx.cx/cgi-bin/blog/index.cgi/geocouch-geospatial-queries-with-couchdb
, and it allows me to mix SQL and couchdb access, all with the couchdb
model and without breaking replication.
Requirement 1: I have a User/Role/Permission model for security. Users
have roles and roles have permissions, and the primary (and very
common) operation is to check for a transitive (user, permission)
pair, which involves a join. It would be wrong to invert the user-role
relationship and put both a list of users and a list of permissions
into the role document because it is the user that joins roles and the
maintenance of the user-role relationship is done when editing the user.
Requirement 2: I have a metadata search mechanism that allows the user
to construct arbitrary queries (in the problem domain, not SQL). The
transformation of the model from relational to document-based has
eliminated any joins, but the problem isn't amenable to the oft-
described key-based virtual joins, and I need to avoid retrieving
potentially large sets of results and doing set operations in the
client because it doesn't scale.
My solution is to use the notification mechanism to maintain
(multiple) SQLite databases containing the document keys I need to
search over. In the URP example, I store (db, src, dest) records. I
also store the last seqno, so that I can do incremental updates to
SQLite.
Then, using _external, I can make queries of SQLite to get me either
(user, permission) pairs using a self-join, or in the case of the
arbitrary metadata queries, a list of document ids.
The primary difficulties I have with this simple model are:
1) The notification mechanism doesn't give me enough information.
Currently I have to do a _all_docs_by_seq and check for deletions by
attempting to get each document, which I have to do for every document
in any case (unless I use transparent id's) to determine if I'm
interested in it, and then get the data. I presume this technique
works because deletion is actually a form of update until compaction
(I copied it from GeoCouch).
** SUGGESTION ** I would prefer an update interface that gave me a
stream of (old, new) document pairs, which covers add/update/delete,
plus a (from, to) seqno pair. Have a 'from' seqno lets me know when I
have to trigger a full rescan, which I need to do in a variety of
circumstances such as configuration change.
2) The process is not as performant as it could be because of the JSON
marshalling.
** SUGGESTION ** I would prefer to write the notification process in
erlang, and be passed the internal representation of the pre/post
documents to avoid marshalling. Same for map/reduce. I know I can
modify couchdb explicitly to do this, but that commits me to git
tracking/merging etc. A in-process plugin option would be better.
3) Allowing the notification and _external handlers to communicate is
more work than it should be. I want to do this so that the _external
handler can cache the queries and invalidate the cache in response to
updates.
As an alternative, for the URP data, I could use memcached and let the
notification process invalidate the cache, which would mean that the
_external handler doesn't have to cache at all because the client
would only hit it on a memcached miss. The ultimate extension of this
would be to have no _external handler, and have the notification
process not use SQLite at all and instead let the client maintain the
data in memcached using normal couchdb requests to calculate the
transitive closure, which would be invalidated by the notification
process. Unless the client passivates this data to disk it would mean
recreating it (on-demand, incrementally) each time the client process
starts. This is a difficult problem with multiple clients (from a
development coordination perspective), so maybe an _external process
is still required, even if it doesn't use on-disk passivation.
That solution doesn't work for the generalized metadata queries, but
that's not a problem.
** SUGGESTION ** I think the _external mechanism (or something better)
should be integrated as a matter of priority. IMO this is a key
mechanism for augmenting the map/reduce model to satisfy more use-cases.
** SUGGESTION ** A plugin mechanism should allow a plugin to both
listen to notifications and respond to e.g. _external.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Human beings, who are almost unique in having the ability to learn
from the experience of others, are also remarkable for their apparent
disinclination to do so.
-- Douglas Adams
Re: Simplest possible plugin/component mechanism
Posted by Mahesh Paolini-Subramanya <ma...@aptela.com>.
Could not agree more. Back when (way back when) we used to do
something similar to extend the NeXTStep widget set. It was usually
functional as long as I was the *only* person doing anything, but
trying to manage/maintain this across multiple users (or heck, just
myself after a 3 month gap) was insane. I eventually gave up on the
whole thing.
Also, i suspect that the security issues might be interesting, to say
the least... :-)
cheers
---
Mahesh Paolini-Subramanya
CTO, Aptela Inc.
(703.386.1500 x9100)
http://www.aptela.com
On Nov 6, 2008, at 12:08 AM, Antony Blakey wrote:
>
> On 06/11/2008, at 3:27 PM, Paul Davis wrote:
>
>> Also, not sure if this is gonna get laughed at or not, but reading
>> the
>> docs on the erl_prim_laoder, it looks like it actually wouldn't be
>> that hard to write a thing that could pull in erlang code from a
>> document. Obviously there'd be some security issues to figure out,
>> but
>> how fucking cool would it be to replicate a design doc and not only
>> have a web app, but add new functionality to the underlying server? I
>> mean, that's one hell of a plugin system if you ask me.
>
> Danger Will Robinson!
>
> In Smalltalk I run into the Subject/Object problem all the time. When
> you require your system to be stable in order to fix a problem with
> the system, you're usually SOL. Imagine deploying a plugin that
> crapped out the very mechanism used to update and deploy plugins. You
> would need a fallback mechanism for deploying plugins (e.g. ssh/rsync
> etc), in which case you might as well make the fallback mechanism the
> primary mechanism and just save yourself the grief of vicious
> circularity.
>
> Antony Blakey
> -------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> There is nothing more difficult to plan, more doubtful of success, nor
> more dangerous to manage than the creation of a new order of things...
> Whenever his enemies have the ability to attack the innovator, they do
> so with the passion of partisans, while the others defend him
> sluggishly, So that the innovator and his party alike are vulnerable.
> -- Niccolo Machiavelli, 1513, The Prince.
>
>
Re: Simplest possible plugin/component mechanism
Posted by Antony Blakey <an...@gmail.com>.
On 06/11/2008, at 3:27 PM, Paul Davis wrote:
> Also, not sure if this is gonna get laughed at or not, but reading the
> docs on the erl_prim_laoder, it looks like it actually wouldn't be
> that hard to write a thing that could pull in erlang code from a
> document. Obviously there'd be some security issues to figure out, but
> how fucking cool would it be to replicate a design doc and not only
> have a web app, but add new functionality to the underlying server? I
> mean, that's one hell of a plugin system if you ask me.
Danger Will Robinson!
In Smalltalk I run into the Subject/Object problem all the time. When
you require your system to be stable in order to fix a problem with
the system, you're usually SOL. Imagine deploying a plugin that
crapped out the very mechanism used to update and deploy plugins. You
would need a fallback mechanism for deploying plugins (e.g. ssh/rsync
etc), in which case you might as well make the fallback mechanism the
primary mechanism and just save yourself the grief of vicious
circularity.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
There is nothing more difficult to plan, more doubtful of success, nor
more dangerous to manage than the creation of a new order of things...
Whenever his enemies have the ability to attack the innovator, they do
so with the passion of partisans, while the others defend him
sluggishly, So that the innovator and his party alike are vulnerable.
-- Niccolo Machiavelli, 1513, The Prince.
Re: Simplest possible plugin/component mechanism
Posted by Antony Blakey <an...@gmail.com>.
On 06/11/2008, at 3:27 PM, Paul Davis wrote:
> Also, not sure if this is gonna get laughed at or not, but reading the
> docs on the erl_prim_laoder, it looks like it actually wouldn't be
> that hard to write a thing that could pull in erlang code from a
> document. Obviously there'd be some security issues to figure out, but
> how fucking cool would it be to replicate a design doc and not only
> have a web app, but add new functionality to the underlying server? I
> mean, that's one hell of a plugin system if you ask me.
Danger Will Robinson!
In Smalltalk I run into the Subject/Object problem all the time. When
you require your system to be stable in order to fix a problem with
the system, you're usually SOL. Imagine deploying a plugin that
crapped out the very mechanism used to update and deploy plugins. You
would need a fallback mechanism for deploying plugins (e.g. ssh/rsync
etc), in which case you might as well make the fallback mechanism the
primary mechanism and just save yourself the grief of vicious
circularity.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
There is nothing more difficult to plan, more doubtful of success, nor
more dangerous to manage than the creation of a new order of things...
Whenever his enemies have the ability to attack the innovator, they do
so with the passion of partisans, while the others defend him
sluggishly, So that the innovator and his party alike are vulnerable.
-- Niccolo Machiavelli, 1513, The Prince.
Re: Simplest possible plugin/component mechanism
Posted by Paul Davis <pa...@gmail.com>.
Noah,
It appears that we may be able to change the module load paths via the
erl_prim_loader module. So having a config option that is has some
list of plugin paths, and then having a standardized directory layout
akin to what Antony described could be awesome.
Also, not sure if this is gonna get laughed at or not, but reading the
docs on the erl_prim_laoder, it looks like it actually wouldn't be
that hard to write a thing that could pull in erlang code from a
document. Obviously there'd be some security issues to figure out, but
how fucking cool would it be to replicate a design doc and not only
have a web app, but add new functionality to the underlying server? I
mean, that's one hell of a plugin system if you ask me.
Also, that could be completely unfeasible and I'm just too tired to notice.
Any other more coherent thoughts?
Paul
On Wed, Nov 5, 2008 at 11:29 PM, Noah Slater <ns...@apache.org> wrote:
> Hey,
>
> This is a nice idea, is there anyway to get Erlang to load modules directly
> without having to modify the path in the CouchDB script? That seems like a much
> cleaner solution to me. With this you could either:
>
> * Add a configuration option that loaded a new configuration file
> * Add new configuration files on the command line
>
> You would also need a way to load custom beam files from the configuration.
>
> Open to suggestions on this.
>
> Best,
>
> --
> Noah Slater, http://bytesexual.org/nslater
>
Re: Simplest possible plugin/component mechanism
Posted by Antony Blakey <an...@gmail.com>.
On 06/11/2008, at 2:59 PM, Noah Slater wrote:
> This is a nice idea, is there anyway to get Erlang to load modules
> directly
> without having to modify the path in the CouchDB script?
Yes, code:add_patha / code:add_pathz allow you to modify the load path
dynamically, so there's no reason why the autoload directories need to
be set in the script. You would need to ensure that the paths in the
config file could be relative.
> That seems like a much
> cleaner solution to me. With this you could either:
>
> * Add a configuration option that loaded a new configuration file
> * Add new configuration files on the command line
You could add new plugins or plugin directories on the command line.
I was looking for a mechanism that didn't require any code.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Always have a vision. Why spend your life making other people’s dreams?
-- Orson Welles (1915-1985)
Re: Simplest possible plugin/component mechanism
Posted by Paul Davis <pa...@gmail.com>.
Noah,
It appears that we may be able to change the module load paths via the
erl_prim_loader module. So having a config option that is has some
list of plugin paths, and then having a standardized directory layout
akin to what Antony described could be awesome.
Also, not sure if this is gonna get laughed at or not, but reading the
docs on the erl_prim_laoder, it looks like it actually wouldn't be
that hard to write a thing that could pull in erlang code from a
document. Obviously there'd be some security issues to figure out, but
how fucking cool would it be to replicate a design doc and not only
have a web app, but add new functionality to the underlying server? I
mean, that's one hell of a plugin system if you ask me.
Also, that could be completely unfeasible and I'm just too tired to notice.
Any other more coherent thoughts?
Paul
On Wed, Nov 5, 2008 at 11:29 PM, Noah Slater <ns...@apache.org> wrote:
> Hey,
>
> This is a nice idea, is there anyway to get Erlang to load modules directly
> without having to modify the path in the CouchDB script? That seems like a much
> cleaner solution to me. With this you could either:
>
> * Add a configuration option that loaded a new configuration file
> * Add new configuration files on the command line
>
> You would also need a way to load custom beam files from the configuration.
>
> Open to suggestions on this.
>
> Best,
>
> --
> Noah Slater, http://bytesexual.org/nslater
>
Re: Simplest possible plugin/component mechanism
Posted by Noah Slater <ns...@apache.org>.
Hey,
This is a nice idea, is there anyway to get Erlang to load modules directly
without having to modify the path in the CouchDB script? That seems like a much
cleaner solution to me. With this you could either:
* Add a configuration option that loaded a new configuration file
* Add new configuration files on the command line
You would also need a way to load custom beam files from the configuration.
Open to suggestions on this.
Best,
--
Noah Slater, http://bytesexual.org/nslater
Re: Simplest possible plugin/component mechanism
Posted by Noah Slater <ns...@apache.org>.
Hey,
This is a nice idea, is there anyway to get Erlang to load modules directly
without having to modify the path in the CouchDB script? That seems like a much
cleaner solution to me. With this you could either:
* Add a configuration option that loaded a new configuration file
* Add new configuration files on the command line
You would also need a way to load custom beam files from the configuration.
Open to suggestions on this.
Best,
--
Noah Slater, http://bytesexual.org/nslater
Simplest possible plugin/component mechanism
Posted by Antony Blakey <an...@gmail.com>.
As part of my efforts to write an erlang plugin I did the following to
the bin/couchdb script:
1. Add the following near the head of the file:
PLUGINS_DIRECTORY=/usr/local/etc/couchdb/plugins
PLUGIN_INI_FILES=
PLUGIN_ERLANG_PATHS=
for plugin in $PLUGINS_DIRECTORY/*; do
if test -d $plugin/ebin; then
PLUGIN_ERLANG_PATHS="$PLUGIN_ERLANG_PATHS $plugin/ebin";
fi
if test -f $plugin/config.ini; then
PLUGIN_INI_FILES="$PLUGIN_INI_FILES $plugin/config.ini";
fi
done
2. Modify the code that accumulates the ini files:
if test -n "$INI_FILES"; then
ini_files="$PLUGIN_INI_FILES $INI_FILES"
else
ini_files="$DEFAULT_INI_FILE $LOCAL_INI_FILE $PLUGIN_INI_FILES"
fi
3. Modify the code that starts couchdb:
command="`/opt/local/bin/icu-config --invoke` \
/opt/local/bin/erl $interactive_option -smp auto -sasl
errlog_type error +K true \
-pa /usr/local/lib/couchdb/erlang/lib/couch-0.9.0a-incubating/
ebin \
/usr/local/lib/couchdb/erlang/lib/mochiweb-r82/ebin \
$PLUGIN_ERLANG_PATHS \
-eval \"application:load(inets)\" \
-eval \"application:load(crypto)\" \
-eval \"application:load(couch)\" \
-eval \"crypto:start()\" \
-eval \"inets:start()\" \
-eval \"couch_server:start([$start_arguments]), receive done -
> done end.\" "
Now I can drop components into /usr/local/etc/couchdb/plugins:
/usr/local/etc/plugins/
00test/
config.ini
ebin/
<blah>.beam
Something like this would seem to be the simplest possible mechanism
that allows separately packaged components.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
What can be done with fewer [assumptions] is done in vain with more
-- William of Ockham (ca. 1285-1349)
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Antony Blakey <an...@gmail.com>.
Thanks Paul.
On 05/11/2008, at 11:09 AM, Paul Davis wrote:
> This is gonna be quick as I'm getting to go watching the election
> coverage
As are we in the rest of the world.
> 1) Pretty sure when you iterate over _all_docs_by_seq, each document
> should have a _deleted=true member if it's been deleted, no reason to
> check the database if it was deleted. That's not to say that there may
> be extra information that could be passed to the update processes. (On
> second read through I'm starting to doubt this now. It could be that
> you end up getting this from the _all_docs. My general thing is read
> 500 or so rows from _all_docs_by_seq and then do a multi-get on those
> 500 docs etc. so the deleted info might come from _all_docs).
Good advice, thanks.
> 2) Currently trunk does have an easy method for registering external
> erlang code easily via the different daemons and httpd handlers. Two
> things though, the startup script needs to have a couple extra
> parameters for telling erlang to load other directories than its own
> lib directory. (Or maybe possible config directives? Not sure on
> erlang's runtime directory path support mechanism)
It's the packaging mechanism (load path etc) that I'm primarily
concerned with.
> Also, there are
> parts of the api that will probably need to be extended for you to get
> all the bells and whistles you're wanting. (ie, an erlang
> udpate_notification handler or some such)
The API available to plugins (as opposed to 'read all the code') is
critical here so that plugins don't get entangled in the innards,
which is not only bad for plugin writers, but also makes evolution
likely to break plugins.
And of course documentation is an absolute requirement for wide
adoption.
> 3) This would be alleviated I think if we flesh out the process for
> adding plugins etc. Your updater/query code would all be in one module
> and have access to everything required.
Yes. And +1 for this to be done sooner rather than later.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Success is not the key to happiness. Happiness is the key to success.
-- Albert Schweitzer
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Antony Blakey <an...@gmail.com>.
On 06/11/2008, at 3:36 PM, Paul Davis wrote:
> Not everyone is willing to pick up erlang just for the ability to play
> with couch in their own particular domain. Also, it allows for
> bridging the gap between disparate systems fairly easily.
Sure, but the alternative is allowing an external process to see a
consistent MVCC snapshot across multiple queries.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
I contend that we are both atheists. I just believe in one fewer god
than you do. When you understand why you dismiss all the other
possible gods, you will understand why I dismiss yours.
--Stephen F Roberts
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Nov 6, 2008 at 12:03 AM, Antony Blakey <an...@gmail.com> wrote:
>
> On 06/11/2008, at 3:15 PM, Paul Davis wrote:
>
>> When we index outside of erlang, we don't have the consistent
>> read-state guarantee if we page through the _all_docs_by_seq view as
>> per usual design pattern. We could read the entire thing view into
>> memory, but that has the obvious not-scalable side effect. Thus by
>> default all external indexers (external == accessing couchdb via http)
>> would have the second form of updating until they reach a steady
>> state. This in deed could lead to race conditions with an indexer
>> never managing to stay quite in sync.
>
> Yet another reason to write a plugin rather than a loosely coupled
> _external, and use a core equivalent to an _all_docs_by_seq iterator that
> gives a consistent snapshot.
>
Not everyone is willing to pick up erlang just for the ability to play
with couch in their own particular domain. Also, it allows for
bridging the gap between disparate systems fairly easily.
>> As for the specifics of mnesia vs. sqlite I couldn't tell you. My
>> guess is that the mnesia integration would kick the crap out of the
>> sqlite integeration seeing as mnesia is part of the core library. But
>> you mentioned GIS stuff and I haven't a clue on mnesia support for
>> such things. Also, mnesia has that whole horizontal scale thing baked
>> in.
>
> I'm not interested in GIS, I just used GeoCouch as an example of an external
> indexer. I think Mnesia will do what I want for indexing.
>
> Thanks,
>
> Antony Blakey
> -------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> The fact that an opinion has been widely held is no evidence whatever that
> it is not utterly absurd.
> -- Bertrand Russell
>
>
>
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Antony Blakey <an...@gmail.com>.
On 06/11/2008, at 3:15 PM, Paul Davis wrote:
> When we index outside of erlang, we don't have the consistent
> read-state guarantee if we page through the _all_docs_by_seq view as
> per usual design pattern. We could read the entire thing view into
> memory, but that has the obvious not-scalable side effect. Thus by
> default all external indexers (external == accessing couchdb via http)
> would have the second form of updating until they reach a steady
> state. This in deed could lead to race conditions with an indexer
> never managing to stay quite in sync.
Yet another reason to write a plugin rather than a loosely coupled
_external, and use a core equivalent to an _all_docs_by_seq iterator
that gives a consistent snapshot.
> As for the specifics of mnesia vs. sqlite I couldn't tell you. My
> guess is that the mnesia integration would kick the crap out of the
> sqlite integeration seeing as mnesia is part of the core library. But
> you mentioned GIS stuff and I haven't a clue on mnesia support for
> such things. Also, mnesia has that whole horizontal scale thing baked
> in.
I'm not interested in GIS, I just used GeoCouch as an example of an
external indexer. I think Mnesia will do what I want for indexing.
Thanks,
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The fact that an opinion has been widely held is no evidence whatever
that it is not utterly absurd.
-- Bertrand Russell
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Jan Lehnardt <ja...@apache.org>.
On Nov 6, 2008, at 05:45, Paul Davis wrote:
> 1. Update notifications should at a bare minimum support DB
> create/update/delete notifications. IIRC, create was missing but was a
> minimal patch. Not sure if its been comited or not.
it's in trunk as of r707076 or Wed Oct 22 06:43:08 2008.
Cheers
Jan
--
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Nov 5, 2008 at 11:12 PM, Antony Blakey <an...@gmail.com> wrote:
>
> On Tue, Nov 4, 2008 at 6:47 PM, Antony Blakey <an...@gmail.com>
> wrote:
>
>> My solution is to use the notification mechanism to maintain (multiple)
>> SQLite databases containing the document keys I need to search over. In
>> the
>> URP example, I store (db, src, dest) records. I also store the last seqno,
>> so that I can do incremental updates to SQLite.
>>
>> Then, using _external, I can make queries of SQLite to get me either
>> (user,
>> permission) pairs using a self-join, or in the case of the arbitrary
>> metadata queries, a list of document ids.
>>
>> The primary difficulties I have with this simple model are:
>>
>> 1) The notification mechanism doesn't give me enough information.
>> Currently
>> I have to do a _all_docs_by_seq and check for deletions by attempting to
>> get
>> each document, which I have to do for every document in any case (unless I
>> use transparent id's) to determine if I'm interested in it, and then get
>> the
>> data. I presume this technique works because deletion is actually a form
>> of
>> update until compaction (I copied it from GeoCouch).
>>
>> ** SUGGESTION ** I would prefer an update interface that gave me a stream
>> of
>> (old, new) document pairs, which covers add/update/delete, plus a (from,
>> to)
>> seqno pair. Have a 'from' seqno lets me know when I have to trigger a full
>> rescan, which I need to do in a variety of circumstances such as
>> configuration change.
>
> In retrospect this is not a good idea. I think notification handlers should
> do nothing more than mark a view or query source as dirty, invalidate a
> cache such as memcached, and possibly check for mods to a config document to
> enable/disable the query service. The _external/plugin query handler should
> do the subsequence processing and update any private data structures or
> on-disk indexes, just as the map/reduce views do, and for the same reason.
> So I don't think the notification mechanism should be changed.
>
> However, that raises a question about external view/query server updating:
> should a view update to the seqnum that is current when the request is
> received, or should it keep looping in an update cycle until the seqnum has
> reached a steady state?
>
> The former would make sense if you just wanted the ensure that the view was
> up-to-date with records a client might have just written in the requesting
> thread, whilst the later would seem to potentially block forever depending
> on the amount of processing required to update the external view and the
> update rate.
>
I'm haven't considered all of the possible ramificaitons of what
information should be presented to update notifications processes, but
my current feelings in my rather tired state are in no particular
order:
1. Update notifications should at a bare minimum support DB
create/update/delete notifications. IIRC, create was missing but was a
minimal patch. Not sure if its been comited or not.
2. View resets may be an addition to the notifications
3. Following from 2, updates to a view may lend credence to having an
update that is "view updated to seq N"
All of those I could see as having particular use cases.
As to updating, if I'm not mistaken, views internally will update to
the latest sequence that was available when the update started.
(Thought being if they access the btree in one go, the consistent read
state would point at them not geting new updates till the next read
request. Not sure if incomming reads durring an update reset this
though)
When we index outside of erlang, we don't have the consistent
read-state guarantee if we page through the _all_docs_by_seq view as
per usual design pattern. We could read the entire thing view into
memory, but that has the obvious not-scalable side effect. Thus by
default all external indexers (external == accessing couchdb via http)
would have the second form of updating until they reach a steady
state. This in deed could lead to race conditions with an indexer
never managing to stay quite in sync.
> Finally, does anyone have advice about the merits of mnesia vs. sqlite (or
> another sql db) for this kind of auxiliary indexing?
>
I'd say it really depends on what you're wanting to accomplish. I for
awhile have contemplated the relative awesome/fail aspects of having
an mnesia layer that treated couchdb as its permanent store and
exported some sort of HTTP query api. I'm pretty sure I've convinced
myself it would be a not-general enough thing that would be worth
supporting. That being said, using it for a specific well defined roll
probably isn't out of the question.
As for the specifics of mnesia vs. sqlite I couldn't tell you. My
guess is that the mnesia integration would kick the crap out of the
sqlite integeration seeing as mnesia is part of the core library. But
you mentioned GIS stuff and I haven't a clue on mnesia support for
such things. Also, mnesia has that whole horizontal scale thing baked
in.
Hopefully that helps more than confounds,
Paul
> Antony Blakey
> --------------------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> Lack of will power has caused more failure than lack of intelligence or
> ability.
> -- Flower A. Newhouse
>
>
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Antony Blakey <an...@gmail.com>.
On Tue, Nov 4, 2008 at 6:47 PM, Antony Blakey
<an...@gmail.com> wrote:
> My solution is to use the notification mechanism to maintain
> (multiple)
> SQLite databases containing the document keys I need to search over.
> In the
> URP example, I store (db, src, dest) records. I also store the last
> seqno,
> so that I can do incremental updates to SQLite.
>
> Then, using _external, I can make queries of SQLite to get me either
> (user,
> permission) pairs using a self-join, or in the case of the arbitrary
> metadata queries, a list of document ids.
>
> The primary difficulties I have with this simple model are:
>
> 1) The notification mechanism doesn't give me enough information.
> Currently
> I have to do a _all_docs_by_seq and check for deletions by
> attempting to get
> each document, which I have to do for every document in any case
> (unless I
> use transparent id's) to determine if I'm interested in it, and then
> get the
> data. I presume this technique works because deletion is actually a
> form of
> update until compaction (I copied it from GeoCouch).
>
> ** SUGGESTION ** I would prefer an update interface that gave me a
> stream of
> (old, new) document pairs, which covers add/update/delete, plus a
> (from, to)
> seqno pair. Have a 'from' seqno lets me know when I have to trigger
> a full
> rescan, which I need to do in a variety of circumstances such as
> configuration change.
In retrospect this is not a good idea. I think notification handlers
should do nothing more than mark a view or query source as dirty,
invalidate a cache such as memcached, and possibly check for mods to a
config document to enable/disable the query service. The _external/
plugin query handler should do the subsequence processing and update
any private data structures or on-disk indexes, just as the map/reduce
views do, and for the same reason. So I don't think the notification
mechanism should be changed.
However, that raises a question about external view/query server
updating: should a view update to the seqnum that is current when the
request is received, or should it keep looping in an update cycle
until the seqnum has reached a steady state?
The former would make sense if you just wanted the ensure that the
view was up-to-date with records a client might have just written in
the requesting thread, whilst the later would seem to potentially
block forever depending on the amount of processing required to update
the external view and the update rate.
Finally, does anyone have advice about the merits of mnesia vs. sqlite
(or another sql db) for this kind of auxiliary indexing?
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Lack of will power has caused more failure than lack of intelligence
or ability.
-- Flower A. Newhouse
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Antony Blakey <an...@gmail.com>.
On Tue, Nov 4, 2008 at 6:47 PM, Antony Blakey
<an...@gmail.com> wrote:
> My solution is to use the notification mechanism to maintain
> (multiple)
> SQLite databases containing the document keys I need to search over.
> In the
> URP example, I store (db, src, dest) records. I also store the last
> seqno,
> so that I can do incremental updates to SQLite.
>
> Then, using _external, I can make queries of SQLite to get me either
> (user,
> permission) pairs using a self-join, or in the case of the arbitrary
> metadata queries, a list of document ids.
>
> The primary difficulties I have with this simple model are:
>
> 1) The notification mechanism doesn't give me enough information.
> Currently
> I have to do a _all_docs_by_seq and check for deletions by
> attempting to get
> each document, which I have to do for every document in any case
> (unless I
> use transparent id's) to determine if I'm interested in it, and then
> get the
> data. I presume this technique works because deletion is actually a
> form of
> update until compaction (I copied it from GeoCouch).
>
> ** SUGGESTION ** I would prefer an update interface that gave me a
> stream of
> (old, new) document pairs, which covers add/update/delete, plus a
> (from, to)
> seqno pair. Have a 'from' seqno lets me know when I have to trigger
> a full
> rescan, which I need to do in a variety of circumstances such as
> configuration change.
In retrospect this is not a good idea. I think notification handlers
should do nothing more than mark a view or query source as dirty,
invalidate a cache such as memcached, and possibly check for mods to a
config document to enable/disable the query service. The _external/
plugin query handler should do the subsequence processing and update
any private data structures or on-disk indexes, just as the map/reduce
views do, and for the same reason. So I don't think the notification
mechanism should be changed.
However, that raises a question about external view/query server
updating: should a view update to the seqnum that is current when the
request is received, or should it keep looping in an update cycle
until the seqnum has reached a steady state?
The former would make sense if you just wanted the ensure that the
view was up-to-date with records a client might have just written in
the requesting thread, whilst the later would seem to potentially
block forever depending on the amount of processing required to update
the external view and the update rate.
Finally, does anyone have advice about the merits of mnesia vs. sqlite
(or another sql db) for this kind of auxiliary indexing?
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Lack of will power has caused more failure than lack of intelligence
or ability.
-- Flower A. Newhouse
Simplest possible plugin/component mechanism
Posted by Antony Blakey <an...@gmail.com>.
As part of my efforts to write an erlang plugin I did the following to
the bin/couchdb script:
1. Add the following near the head of the file:
PLUGINS_DIRECTORY=/usr/local/etc/couchdb/plugins
PLUGIN_INI_FILES=
PLUGIN_ERLANG_PATHS=
for plugin in $PLUGINS_DIRECTORY/*; do
if test -d $plugin/ebin; then
PLUGIN_ERLANG_PATHS="$PLUGIN_ERLANG_PATHS $plugin/ebin";
fi
if test -f $plugin/config.ini; then
PLUGIN_INI_FILES="$PLUGIN_INI_FILES $plugin/config.ini";
fi
done
2. Modify the code that accumulates the ini files:
if test -n "$INI_FILES"; then
ini_files="$PLUGIN_INI_FILES $INI_FILES"
else
ini_files="$DEFAULT_INI_FILE $LOCAL_INI_FILE $PLUGIN_INI_FILES"
fi
3. Modify the code that starts couchdb:
command="`/opt/local/bin/icu-config --invoke` \
/opt/local/bin/erl $interactive_option -smp auto -sasl
errlog_type error +K true \
-pa /usr/local/lib/couchdb/erlang/lib/couch-0.9.0a-incubating/
ebin \
/usr/local/lib/couchdb/erlang/lib/mochiweb-r82/ebin \
$PLUGIN_ERLANG_PATHS \
-eval \"application:load(inets)\" \
-eval \"application:load(crypto)\" \
-eval \"application:load(couch)\" \
-eval \"crypto:start()\" \
-eval \"inets:start()\" \
-eval \"couch_server:start([$start_arguments]), receive done -
> done end.\" "
Now I can drop components into /usr/local/etc/couchdb/plugins:
/usr/local/etc/plugins/
00test/
config.ini
ebin/
<blah>.beam
Something like this would seem to be the simplest possible mechanism
that allows separately packaged components.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
What can be done with fewer [assumptions] is done in vain with more
-- William of Ockham (ca. 1285-1349)
Re: Architectural advice requested, and some 1.0 suggestions.
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Nov 4, 2008 at 6:47 PM, Antony Blakey <an...@gmail.com> wrote:
> Hi, [ I'm posting to both -user and -dev because IMO it's relevant to both ]
>
> I'm in the middle of a trial port of a system from postgresql to couchdb.
> The system has a distribution/replication requirement due primarily to
> unreliable connectivity and limited end-to-end bandwidth (and high latency)
> over a geographically dispersed set of users.
>
> I have two requirements that don't fit well with the basic couchdb model.
> I've developed solutions for which I'd appreciate some feedback. Also it's a
> request for the notification mechanism to be improved, something like
> _external to moved to the core, and/or an in-process erlang plugin model
> that could replace notification/_external-based techniques.
>
> My solution is a generalization of the GeoCouch technique:
> http://vmx.cx/cgi-bin/blog/index.cgi/geocouch-geospatial-queries-with-couchdb,
> and it allows me to mix SQL and couchdb access, all with the couchdb model
> and without breaking replication.
>
> Requirement 1: I have a User/Role/Permission model for security. Users have
> roles and roles have permissions, and the primary (and very common)
> operation is to check for a transitive (user, permission) pair, which
> involves a join. It would be wrong to invert the user-role relationship and
> put both a list of users and a list of permissions into the role document
> because it is the user that joins roles and the maintenance of the user-role
> relationship is done when editing the user.
>
> Requirement 2: I have a metadata search mechanism that allows the user to
> construct arbitrary queries (in the problem domain, not SQL). The
> transformation of the model from relational to document-based has eliminated
> any joins, but the problem isn't amenable to the oft-described key-based
> virtual joins, and I need to avoid retrieving potentially large sets of
> results and doing set operations in the client because it doesn't scale.
>
> My solution is to use the notification mechanism to maintain (multiple)
> SQLite databases containing the document keys I need to search over. In the
> URP example, I store (db, src, dest) records. I also store the last seqno,
> so that I can do incremental updates to SQLite.
>
> Then, using _external, I can make queries of SQLite to get me either (user,
> permission) pairs using a self-join, or in the case of the arbitrary
> metadata queries, a list of document ids.
>
> The primary difficulties I have with this simple model are:
>
> 1) The notification mechanism doesn't give me enough information. Currently
> I have to do a _all_docs_by_seq and check for deletions by attempting to get
> each document, which I have to do for every document in any case (unless I
> use transparent id's) to determine if I'm interested in it, and then get the
> data. I presume this technique works because deletion is actually a form of
> update until compaction (I copied it from GeoCouch).
>
> ** SUGGESTION ** I would prefer an update interface that gave me a stream of
> (old, new) document pairs, which covers add/update/delete, plus a (from, to)
> seqno pair. Have a 'from' seqno lets me know when I have to trigger a full
> rescan, which I need to do in a variety of circumstances such as
> configuration change.
>
> 2) The process is not as performant as it could be because of the JSON
> marshalling.
>
> ** SUGGESTION ** I would prefer to write the notification process in erlang,
> and be passed the internal representation of the pre/post documents to avoid
> marshalling. Same for map/reduce. I know I can modify couchdb explicitly to
> do this, but that commits me to git tracking/merging etc. A in-process
> plugin option would be better.
>
> 3) Allowing the notification and _external handlers to communicate is more
> work than it should be. I want to do this so that the _external handler can
> cache the queries and invalidate the cache in response to updates.
>
> As an alternative, for the URP data, I could use memcached and let the
> notification process invalidate the cache, which would mean that the
> _external handler doesn't have to cache at all because the client would only
> hit it on a memcached miss. The ultimate extension of this would be to have
> no _external handler, and have the notification process not use SQLite at
> all and instead let the client maintain the data in memcached using normal
> couchdb requests to calculate the transitive closure, which would be
> invalidated by the notification process. Unless the client passivates this
> data to disk it would mean recreating it (on-demand, incrementally) each
> time the client process starts. This is a difficult problem with multiple
> clients (from a development coordination perspective), so maybe an _external
> process is still required, even if it doesn't use on-disk passivation.
>
> That solution doesn't work for the generalized metadata queries, but that's
> not a problem.
>
> ** SUGGESTION ** I think the _external mechanism (or something better)
> should be integrated as a matter of priority. IMO this is a key mechanism
> for augmenting the map/reduce model to satisfy more use-cases.
>
> ** SUGGESTION ** A plugin mechanism should allow a plugin to both listen to
> notifications and respond to e.g. _external.
>
> Antony Blakey
> --------------------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> Human beings, who are almost unique in having the ability to learn from the
> experience of others, are also remarkable for their apparent disinclination
> to do so.
> -- Douglas Adams
>
>
>
This is gonna be quick as I'm getting to go watching the election
coverage, so lets get started:
1) Pretty sure when you iterate over _all_docs_by_seq, each document
should have a _deleted=true member if it's been deleted, no reason to
check the database if it was deleted. That's not to say that there may
be extra information that could be passed to the update processes. (On
second read through I'm starting to doubt this now. It could be that
you end up getting this from the _all_docs. My general thing is read
500 or so rows from _all_docs_by_seq and then do a multi-get on those
500 docs etc. so the deleted info might come from _all_docs).
2) Currently trunk does have an easy method for registering external
erlang code easily via the different daemons and httpd handlers. Two
things though, the startup script needs to have a couple extra
parameters for telling erlang to load other directories than its own
lib directory. (Or maybe possible config directives? Not sure on
erlang's runtime directory path support mechanism) Also, there are
parts of the api that will probably need to be extended for you to get
all the bells and whistles you're wanting. (ie, an erlang
udpate_notification handler or some such)
3) This would be alleviated I think if we flesh out the process for
adding plugins etc. Your updater/query code would all be in one module
and have access to everything required.
Those are my thoughts for now. I'll think more on it later.
HTH,
Paul