Re: how to do http request within server-side _show or _view functions?

[Scott Shumaker <ss...@gmail.com>]

Here's what we do:

All of our writes go through our backend (the clients don't directly
talk to couchdb).
Whenever we're writing, we fetch the user's permissions object out of
the db (typically, it's actually available in the client-side cookies
in an encrypted fashion, so we save the fetch).   Ideally, the
validate_doc_update would allow you to check incoming HTTP headers as
well - in case you don't want to use CouchDB's built in user
authentication, but still want your own validation to run.   That's
not currently the case, so instead we stick a copy of the user's
permission object into the object we're about to write to couch, and
then write it.  Then the validate_doc_update has access to the
permissions as a member of the object that is being written.

Not ideal, but it works.  I'd love for validate_doc_update to take
HTTP headers - especially an additional JSON parameter, like a custom
userCtx.

On Thu, Apr 23, 2009 at 1:10 PM, Samuel Wan <sa...@samuelwan.com> wrote:
> Thanks Robert. I realized I jumped too far ahead and I'm asking the
> wrong question. At what point should a developer write middleware for
> authentication and authorization instead of relying on CouchDB's
> facilities?
>
> Authentication:
> If authentication is enabled on CouchDB, the user submits
> username/password at the basic auth prompt. These credentials go to
> the authentication_handler specified by the default.ini file. If you
> store credentials as CouchDB documents, could you theoretically use
> the null_authentication_handler to allow all requests, and then use
> validate_doc_update to authenticate the userCtx object against
> credential documents in the database? Is this not possible because
> HTTP requests aren't supported on the server?
>
> Authorization:
> If you store permissions as CouchDB documents, how would you use
> validate_doc_update to compare userCtx or newDoc against the stored
> permissions document?  The validate_doc_update function cannot make a
> separate HTTP request to retrieve another permissions document stored
> in CouchDB.
>
> I'm basing these questions on the technical overview page
> (http://couchdb.apache.org/docs/overview.html), which says: "A basic
> “author only” update document model is trivial to implement, where
> document updates are validated to check if the user is listed in an
> “author” field in the existing document. More dynamic models are also
> possible, like checking a separate user account profile for permission
> settings."
>
> The security_validation.js test will check an "author" document
> property for authorization, but the test seems to presume that the
> user has already been authenticated because the validate_doc_update
> function expects a userCtx object. The tests don't hint at how more
> dynamic models are possible.
>
> Any hints would be greatly appreciated.
>
> -Sam
>
> On Thu, Apr 23, 2009 at 10:14 AM, Robert Newson <ro...@gmail.com> wrote:
>> I think the same rule applies, you should use the parameters supplied,
>> and only those, to implement your validate function.
>>
>> You have the old document, the new document and the user context
>> object to play with, it sounds like that's enough for your case
>> anyway?
>>
>> B.
>>
>> On Thu, Apr 23, 2009 at 5:59 PM, Samuel Wan <sa...@samuelwan.com> wrote:
>>> Thanks. What about validate_doc_update? My goal was to check a hashed
>>> key in the document submitted by a user by comparing the key with a
>>> stored document when the validate_doc_update is called.
>>>
>>> -Sam
>>>
>>> On Thu, Apr 23, 2009 at 9:33 AM, Robert Newson <ro...@gmail.com> wrote:
>>>> I was just reading this (at
>>>> http://wiki.apache.org/couchdb/Formatting_with_Show_and_List);
>>>>
>>>> "Show and list functions are side effect free and idempotent. *They
>>>> can not make additional HTTP requests against CouchDB*. Their purpose
>>>> is to render JSON documents in other formats."
>>>>
>>>> So they can't, and it's intentional.
>>>>
>>>> B.
>>>>
>>>> On Thu, Apr 23, 2009 at 4:38 PM, Samuel Wan <sa...@samuelwan.com> wrote:
>>>>> Is there a way to do an http request within the server-side _show or
>>>>> _view functions, or is the JS context limited to what's defined by
>>>>> Spidermonkey and main.js? My understanding is that Spidermonkey
>>>>> doesn't provide an xmlhttprequest object.
>>>>>
>>>>> Also, does CouchDB ship with an interactive javascript shell, or do
>>>>> you have to build and install your own? I found the couchjs
>>>>> executable, but it only interprets javascript files.
>>>>>
>>>>> -Sam
>>>>>
>>>>
>>>
>>
>

Re: how to do http request within server-side _show or _view functions?

[Chris Anderson <jc...@apache.org>]

On Thu, May 21, 2009 at 5:06 PM, Scott Shumaker <ss...@gmail.com> wrote:
> I'm sure it will be a while before we move that direction.  One
> alternative is to think about making CouchDB's storage engine
> pluggable.  For example, with a moderate amount of work, you could use
> an existing distributed, fast key-value store like Project Voldemort
> or Scalaris to store the actual objects.  The authentication,
> versioning, and view architecture could sit on top of that.  Project
> Voldemort supports custom serialization types, so you could even store
> binary-rep erlang terms in it.
>

Well now that sounds like a mashup I'd love to see. Someone please
hack Couch to store keys in other stores. More power to ya!

> Scott
>
> On Wed, May 20, 2009 at 12:53 PM, Paul Davis
> <pa...@gmail.com> wrote:
>> On Wed, May 20, 2009 at 3:19 PM, Scott Shumaker <ss...@gmail.com> wrote:
>>> Apparently at the cost of limiting the functionality of dozens of
>>> unrelated features - everything from transactions to validation.
>>>
>>> I grow ever more convinced that CouchDB's replication is an albatross.
>>
>> An albatross is only bad once you kill it. :)
>>
>>>  It's analogous to RAID-1 - easy to code but pretty limited (and now
>>> very outdated) tech.  It wouldn't be a problem if it were an isolated
>>> feature, but it hamstrings so many others.
>>>
>>> Maybe it makes sense for a particular kind of project, but for most
>>> real-world applications you can get better (more scalable, higher
>>> performant, and more reliable) results with some sort of Paxos or
>>> Vector-clock architecture with multiple committers and readers.  I do
>>> hope to see CouchDB move in that direction - and I think it has a good
>>> chance of becoming the web persistent storage engine of choice if it
>>> does.
>>>
>>
>> If you really want to see CouchDB move towards a replication mechanism
>> that's more complicated than the current model, its going to take no
>> less than working code. I've seen some literature that leads me to
>> believe that we could extend the current model to add more operational
>> modes (and allow for more advanced features in other places). Though
>> in my priority queue this ranks somewhere around "would be
>> interesting" and thus is barely a blip on the radar considering all
>> the other work that needs to be done.
>>
>> HTH,
>> Paul Davis
>>
>>> Scott
>>>
>>> On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>>
>>>> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>>>>
>>>>> Why are the validation functions run at replication time?
>>>>
>>>> As far as the data store in CouchDB is concerned, replication is
>>>> just another client doing GETs & PPSTs). Replication uses the
>>>> same API any user faces.
>>>>
>>>> Cheers
>>>> Jan
>>>> --
>>>>
>>>>
>>>>>
>>>>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>>>>
>>>>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>>>>> userCtx.
>>>>>>>
>>>>>>
>>>>>> The problem with this approach is that the validation functions are
>>>>>> run at replication time as well as at initial update time. This is why
>>>>>> we need to abstract the http information into a userCtx, because the
>>>>>> full request object won't be available for replay later (and the
>>>>>> replicating userCtx is used at rep time, not the original Ctx, so
>>>>>> replay wouldn't be appropriate anyway.)
>>>>>>
>>>>>> Both concerns in this thread (custom http auth & using user creds from
>>>>>> a db) are best addressed by writing another authentication handler or
>>>>>> two. I believe there is work underway to do this but I'm not sure the
>>>>>> current state.
>>>>>>
>>>>>> We are very open to patches in the auth handler section of the code.
>>>>>> Please inquire on dev@ (or drop a patch on
>>>>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>>>>> here.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chris Anderson
>>>>>> http://jchrisa.net
>>>>>> http://couch.io
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: how to do http request within server-side _show or _view functions?

[Scott Shumaker <ss...@gmail.com>]

I'm sure it will be a while before we move that direction.  One
alternative is to think about making CouchDB's storage engine
pluggable.  For example, with a moderate amount of work, you could use
an existing distributed, fast key-value store like Project Voldemort
or Scalaris to store the actual objects.  The authentication,
versioning, and view architecture could sit on top of that.  Project
Voldemort supports custom serialization types, so you could even store
binary-rep erlang terms in it.

Scott

On Wed, May 20, 2009 at 12:53 PM, Paul Davis
<pa...@gmail.com> wrote:
> On Wed, May 20, 2009 at 3:19 PM, Scott Shumaker <ss...@gmail.com> wrote:
>> Apparently at the cost of limiting the functionality of dozens of
>> unrelated features - everything from transactions to validation.
>>
>> I grow ever more convinced that CouchDB's replication is an albatross.
>
> An albatross is only bad once you kill it. :)
>
>>  It's analogous to RAID-1 - easy to code but pretty limited (and now
>> very outdated) tech.  It wouldn't be a problem if it were an isolated
>> feature, but it hamstrings so many others.
>>
>> Maybe it makes sense for a particular kind of project, but for most
>> real-world applications you can get better (more scalable, higher
>> performant, and more reliable) results with some sort of Paxos or
>> Vector-clock architecture with multiple committers and readers.  I do
>> hope to see CouchDB move in that direction - and I think it has a good
>> chance of becoming the web persistent storage engine of choice if it
>> does.
>>
>
> If you really want to see CouchDB move towards a replication mechanism
> that's more complicated than the current model, its going to take no
> less than working code. I've seen some literature that leads me to
> believe that we could extend the current model to add more operational
> modes (and allow for more advanced features in other places). Though
> in my priority queue this ranks somewhere around "would be
> interesting" and thus is barely a blip on the radar considering all
> the other work that needs to be done.
>
> HTH,
> Paul Davis
>
>> Scott
>>
>> On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>>>
>>>> Why are the validation functions run at replication time?
>>>
>>> As far as the data store in CouchDB is concerned, replication is
>>> just another client doing GETs & PPSTs). Replication uses the
>>> same API any user faces.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
>>>>
>>>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>>>
>>>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>>>> userCtx.
>>>>>>
>>>>>
>>>>> The problem with this approach is that the validation functions are
>>>>> run at replication time as well as at initial update time. This is why
>>>>> we need to abstract the http information into a userCtx, because the
>>>>> full request object won't be available for replay later (and the
>>>>> replicating userCtx is used at rep time, not the original Ctx, so
>>>>> replay wouldn't be appropriate anyway.)
>>>>>
>>>>> Both concerns in this thread (custom http auth & using user creds from
>>>>> a db) are best addressed by writing another authentication handler or
>>>>> two. I believe there is work underway to do this but I'm not sure the
>>>>> current state.
>>>>>
>>>>> We are very open to patches in the auth handler section of the code.
>>>>> Please inquire on dev@ (or drop a patch on
>>>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>>>> here.
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chris Anderson
>>>>> http://jchrisa.net
>>>>> http://couch.io
>>>>>
>>>>
>>>
>>>
>>
>

Re: how to do http request within server-side _show or _view functions?

[Paul Davis <pa...@gmail.com>]

On Wed, May 20, 2009 at 3:19 PM, Scott Shumaker <ss...@gmail.com> wrote:
> Apparently at the cost of limiting the functionality of dozens of
> unrelated features - everything from transactions to validation.
>
> I grow ever more convinced that CouchDB's replication is an albatross.

An albatross is only bad once you kill it. :)

>  It's analogous to RAID-1 - easy to code but pretty limited (and now
> very outdated) tech.  It wouldn't be a problem if it were an isolated
> feature, but it hamstrings so many others.
>
> Maybe it makes sense for a particular kind of project, but for most
> real-world applications you can get better (more scalable, higher
> performant, and more reliable) results with some sort of Paxos or
> Vector-clock architecture with multiple committers and readers.  I do
> hope to see CouchDB move in that direction - and I think it has a good
> chance of becoming the web persistent storage engine of choice if it
> does.
>

If you really want to see CouchDB move towards a replication mechanism
that's more complicated than the current model, its going to take no
less than working code. I've seen some literature that leads me to
believe that we could extend the current model to add more operational
modes (and allow for more advanced features in other places). Though
in my priority queue this ranks somewhere around "would be
interesting" and thus is barely a blip on the radar considering all
the other work that needs to be done.

HTH,
Paul Davis

> Scott
>
> On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>>
>>> Why are the validation functions run at replication time?
>>
>> As far as the data store in CouchDB is concerned, replication is
>> just another client doing GETs & PPSTs). Replication uses the
>> same API any user faces.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>>
>>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>>
>>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>>> userCtx.
>>>>>
>>>>
>>>> The problem with this approach is that the validation functions are
>>>> run at replication time as well as at initial update time. This is why
>>>> we need to abstract the http information into a userCtx, because the
>>>> full request object won't be available for replay later (and the
>>>> replicating userCtx is used at rep time, not the original Ctx, so
>>>> replay wouldn't be appropriate anyway.)
>>>>
>>>> Both concerns in this thread (custom http auth & using user creds from
>>>> a db) are best addressed by writing another authentication handler or
>>>> two. I believe there is work underway to do this but I'm not sure the
>>>> current state.
>>>>
>>>> We are very open to patches in the auth handler section of the code.
>>>> Please inquire on dev@ (or drop a patch on
>>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>>> here.
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>> --
>>>> Chris Anderson
>>>> http://jchrisa.net
>>>> http://couch.io
>>>>
>>>
>>
>>
>

Re: how to do http request within server-side _show or _view functions?

[Scott Shumaker <ss...@gmail.com>]

Apparently at the cost of limiting the functionality of dozens of
unrelated features - everything from transactions to validation.

I grow ever more convinced that CouchDB's replication is an albatross.
 It's analogous to RAID-1 - easy to code but pretty limited (and now
very outdated) tech.  It wouldn't be a problem if it were an isolated
feature, but it hamstrings so many others.

Maybe it makes sense for a particular kind of project, but for most
real-world applications you can get better (more scalable, higher
performant, and more reliable) results with some sort of Paxos or
Vector-clock architecture with multiple committers and readers.  I do
hope to see CouchDB move in that direction - and I think it has a good
chance of becoming the web persistent storage engine of choice if it
does.

Scott

On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>
>> Why are the validation functions run at replication time?
>
> As far as the data store in CouchDB is concerned, replication is
> just another client doing GETs & PPSTs). Replication uses the
> same API any user faces.
>
> Cheers
> Jan
> --
>
>
>>
>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>
>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>> wrote:
>>>>
>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>> userCtx.
>>>>
>>>
>>> The problem with this approach is that the validation functions are
>>> run at replication time as well as at initial update time. This is why
>>> we need to abstract the http information into a userCtx, because the
>>> full request object won't be available for replay later (and the
>>> replicating userCtx is used at rep time, not the original Ctx, so
>>> replay wouldn't be appropriate anyway.)
>>>
>>> Both concerns in this thread (custom http auth & using user creds from
>>> a db) are best addressed by writing another authentication handler or
>>> two. I believe there is work underway to do this but I'm not sure the
>>> current state.
>>>
>>> We are very open to patches in the auth handler section of the code.
>>> Please inquire on dev@ (or drop a patch on
>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>> here.
>>>
>>> Chris
>>>
>>>
>>>
>>> --
>>> Chris Anderson
>>> http://jchrisa.net
>>> http://couch.io
>>>
>>
>
>

Re: how to do http request within server-side _show or _view functions?

[Jan Lehnardt <ja...@apache.org>]

On 17 May 2009, at 01:28, Scott Shumaker wrote:

> Why are the validation functions run at replication time?

As far as the data store in CouchDB is concerned, replication is
just another client doing GETs & PPSTs). Replication uses the
same API any user faces.

Cheers
Jan
--


>
> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org>  
> wrote:
>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker  
>> <ss...@gmail.com> wrote:
>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>> HTTP headers - especially an additional JSON parameter, like a  
>>> custom
>>> userCtx.
>>>
>>
>> The problem with this approach is that the validation functions are
>> run at replication time as well as at initial update time. This is  
>> why
>> we need to abstract the http information into a userCtx, because the
>> full request object won't be available for replay later (and the
>> replicating userCtx is used at rep time, not the original Ctx, so
>> replay wouldn't be appropriate anyway.)
>>
>> Both concerns in this thread (custom http auth & using user creds  
>> from
>> a db) are best addressed by writing another authentication handler or
>> two. I believe there is work underway to do this but I'm not sure the
>> current state.
>>
>> We are very open to patches in the auth handler section of the code.
>> Please inquire on dev@ (or drop a patch on
>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>> here.
>>
>> Chris
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
>

Re: how to do http request within server-side _show or _view functions?

[Scott Shumaker <ss...@gmail.com>]

Why are the validation functions run at replication time?

On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com> wrote:
>> Not ideal, but it works.  I'd love for validate_doc_update to take
>> HTTP headers - especially an additional JSON parameter, like a custom
>> userCtx.
>>
>
> The problem with this approach is that the validation functions are
> run at replication time as well as at initial update time. This is why
> we need to abstract the http information into a userCtx, because the
> full request object won't be available for replay later (and the
> replicating userCtx is used at rep time, not the original Ctx, so
> replay wouldn't be appropriate anyway.)
>
> Both concerns in this thread (custom http auth & using user creds from
> a db) are best addressed by writing another authentication handler or
> two. I believe there is work underway to do this but I'm not sure the
> current state.
>
> We are very open to patches in the auth handler section of the code.
> Please inquire on dev@ (or drop a patch on
> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
> here.
>
> Chris
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>

Re: how to do http request within server-side _show or _view functions?

[Chris Anderson <jc...@apache.org>]

On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com> wrote:
> Not ideal, but it works.  I'd love for validate_doc_update to take
> HTTP headers - especially an additional JSON parameter, like a custom
> userCtx.
>

The problem with this approach is that the validation functions are
run at replication time as well as at initial update time. This is why
we need to abstract the http information into a userCtx, because the
full request object won't be available for replay later (and the
replicating userCtx is used at rep time, not the original Ctx, so
replay wouldn't be appropriate anyway.)

Both concerns in this thread (custom http auth & using user creds from
a db) are best addressed by writing another authentication handler or
two. I believe there is work underway to do this but I'm not sure the
current state.

We are very open to patches in the auth handler section of the code.
Please inquire on dev@ (or drop a patch on
https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
here.

Chris



-- 
Chris Anderson
http://jchrisa.net
http://couch.io