Re: how to do http request within server-side _show or _view functions?

[Jan Lehnardt <ja...@apache.org>]

On 17 May 2009, at 01:28, Scott Shumaker wrote:

> Why are the validation functions run at replication time?

As far as the data store in CouchDB is concerned, replication is
just another client doing GETs & PPSTs). Replication uses the
same API any user faces.

Cheers
Jan
--


>
> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org>  
> wrote:
>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker  
>> <ss...@gmail.com> wrote:
>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>> HTTP headers - especially an additional JSON parameter, like a  
>>> custom
>>> userCtx.
>>>
>>
>> The problem with this approach is that the validation functions are
>> run at replication time as well as at initial update time. This is  
>> why
>> we need to abstract the http information into a userCtx, because the
>> full request object won't be available for replay later (and the
>> replicating userCtx is used at rep time, not the original Ctx, so
>> replay wouldn't be appropriate anyway.)
>>
>> Both concerns in this thread (custom http auth & using user creds  
>> from
>> a db) are best addressed by writing another authentication handler or
>> two. I believe there is work underway to do this but I'm not sure the
>> current state.
>>
>> We are very open to patches in the auth handler section of the code.
>> Please inquire on dev@ (or drop a patch on
>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>> here.
>>
>> Chris
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
>

Re: how to do http request within server-side _show or _view functions?

[Chris Anderson <jc...@apache.org>]

On Thu, May 21, 2009 at 5:06 PM, Scott Shumaker <ss...@gmail.com> wrote:
> I'm sure it will be a while before we move that direction.  One
> alternative is to think about making CouchDB's storage engine
> pluggable.  For example, with a moderate amount of work, you could use
> an existing distributed, fast key-value store like Project Voldemort
> or Scalaris to store the actual objects.  The authentication,
> versioning, and view architecture could sit on top of that.  Project
> Voldemort supports custom serialization types, so you could even store
> binary-rep erlang terms in it.
>

Well now that sounds like a mashup I'd love to see. Someone please
hack Couch to store keys in other stores. More power to ya!

> Scott
>
> On Wed, May 20, 2009 at 12:53 PM, Paul Davis
> <pa...@gmail.com> wrote:
>> On Wed, May 20, 2009 at 3:19 PM, Scott Shumaker <ss...@gmail.com> wrote:
>>> Apparently at the cost of limiting the functionality of dozens of
>>> unrelated features - everything from transactions to validation.
>>>
>>> I grow ever more convinced that CouchDB's replication is an albatross.
>>
>> An albatross is only bad once you kill it. :)
>>
>>>  It's analogous to RAID-1 - easy to code but pretty limited (and now
>>> very outdated) tech.  It wouldn't be a problem if it were an isolated
>>> feature, but it hamstrings so many others.
>>>
>>> Maybe it makes sense for a particular kind of project, but for most
>>> real-world applications you can get better (more scalable, higher
>>> performant, and more reliable) results with some sort of Paxos or
>>> Vector-clock architecture with multiple committers and readers.  I do
>>> hope to see CouchDB move in that direction - and I think it has a good
>>> chance of becoming the web persistent storage engine of choice if it
>>> does.
>>>
>>
>> If you really want to see CouchDB move towards a replication mechanism
>> that's more complicated than the current model, its going to take no
>> less than working code. I've seen some literature that leads me to
>> believe that we could extend the current model to add more operational
>> modes (and allow for more advanced features in other places). Though
>> in my priority queue this ranks somewhere around "would be
>> interesting" and thus is barely a blip on the radar considering all
>> the other work that needs to be done.
>>
>> HTH,
>> Paul Davis
>>
>>> Scott
>>>
>>> On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>>
>>>> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>>>>
>>>>> Why are the validation functions run at replication time?
>>>>
>>>> As far as the data store in CouchDB is concerned, replication is
>>>> just another client doing GETs & PPSTs). Replication uses the
>>>> same API any user faces.
>>>>
>>>> Cheers
>>>> Jan
>>>> --
>>>>
>>>>
>>>>>
>>>>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>>>>
>>>>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>>>>> userCtx.
>>>>>>>
>>>>>>
>>>>>> The problem with this approach is that the validation functions are
>>>>>> run at replication time as well as at initial update time. This is why
>>>>>> we need to abstract the http information into a userCtx, because the
>>>>>> full request object won't be available for replay later (and the
>>>>>> replicating userCtx is used at rep time, not the original Ctx, so
>>>>>> replay wouldn't be appropriate anyway.)
>>>>>>
>>>>>> Both concerns in this thread (custom http auth & using user creds from
>>>>>> a db) are best addressed by writing another authentication handler or
>>>>>> two. I believe there is work underway to do this but I'm not sure the
>>>>>> current state.
>>>>>>
>>>>>> We are very open to patches in the auth handler section of the code.
>>>>>> Please inquire on dev@ (or drop a patch on
>>>>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>>>>> here.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chris Anderson
>>>>>> http://jchrisa.net
>>>>>> http://couch.io
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: how to do http request within server-side _show or _view functions?

[Scott Shumaker <ss...@gmail.com>]

I'm sure it will be a while before we move that direction.  One
alternative is to think about making CouchDB's storage engine
pluggable.  For example, with a moderate amount of work, you could use
an existing distributed, fast key-value store like Project Voldemort
or Scalaris to store the actual objects.  The authentication,
versioning, and view architecture could sit on top of that.  Project
Voldemort supports custom serialization types, so you could even store
binary-rep erlang terms in it.

Scott

On Wed, May 20, 2009 at 12:53 PM, Paul Davis
<pa...@gmail.com> wrote:
> On Wed, May 20, 2009 at 3:19 PM, Scott Shumaker <ss...@gmail.com> wrote:
>> Apparently at the cost of limiting the functionality of dozens of
>> unrelated features - everything from transactions to validation.
>>
>> I grow ever more convinced that CouchDB's replication is an albatross.
>
> An albatross is only bad once you kill it. :)
>
>>  It's analogous to RAID-1 - easy to code but pretty limited (and now
>> very outdated) tech.  It wouldn't be a problem if it were an isolated
>> feature, but it hamstrings so many others.
>>
>> Maybe it makes sense for a particular kind of project, but for most
>> real-world applications you can get better (more scalable, higher
>> performant, and more reliable) results with some sort of Paxos or
>> Vector-clock architecture with multiple committers and readers.  I do
>> hope to see CouchDB move in that direction - and I think it has a good
>> chance of becoming the web persistent storage engine of choice if it
>> does.
>>
>
> If you really want to see CouchDB move towards a replication mechanism
> that's more complicated than the current model, its going to take no
> less than working code. I've seen some literature that leads me to
> believe that we could extend the current model to add more operational
> modes (and allow for more advanced features in other places). Though
> in my priority queue this ranks somewhere around "would be
> interesting" and thus is barely a blip on the radar considering all
> the other work that needs to be done.
>
> HTH,
> Paul Davis
>
>> Scott
>>
>> On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>>>
>>>> Why are the validation functions run at replication time?
>>>
>>> As far as the data store in CouchDB is concerned, replication is
>>> just another client doing GETs & PPSTs). Replication uses the
>>> same API any user faces.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
>>>>
>>>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>>>
>>>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>>>> userCtx.
>>>>>>
>>>>>
>>>>> The problem with this approach is that the validation functions are
>>>>> run at replication time as well as at initial update time. This is why
>>>>> we need to abstract the http information into a userCtx, because the
>>>>> full request object won't be available for replay later (and the
>>>>> replicating userCtx is used at rep time, not the original Ctx, so
>>>>> replay wouldn't be appropriate anyway.)
>>>>>
>>>>> Both concerns in this thread (custom http auth & using user creds from
>>>>> a db) are best addressed by writing another authentication handler or
>>>>> two. I believe there is work underway to do this but I'm not sure the
>>>>> current state.
>>>>>
>>>>> We are very open to patches in the auth handler section of the code.
>>>>> Please inquire on dev@ (or drop a patch on
>>>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>>>> here.
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chris Anderson
>>>>> http://jchrisa.net
>>>>> http://couch.io
>>>>>
>>>>
>>>
>>>
>>
>

Re: how to do http request within server-side _show or _view functions?

[Paul Davis <pa...@gmail.com>]

On Wed, May 20, 2009 at 3:19 PM, Scott Shumaker <ss...@gmail.com> wrote:
> Apparently at the cost of limiting the functionality of dozens of
> unrelated features - everything from transactions to validation.
>
> I grow ever more convinced that CouchDB's replication is an albatross.

An albatross is only bad once you kill it. :)

>  It's analogous to RAID-1 - easy to code but pretty limited (and now
> very outdated) tech.  It wouldn't be a problem if it were an isolated
> feature, but it hamstrings so many others.
>
> Maybe it makes sense for a particular kind of project, but for most
> real-world applications you can get better (more scalable, higher
> performant, and more reliable) results with some sort of Paxos or
> Vector-clock architecture with multiple committers and readers.  I do
> hope to see CouchDB move in that direction - and I think it has a good
> chance of becoming the web persistent storage engine of choice if it
> does.
>

If you really want to see CouchDB move towards a replication mechanism
that's more complicated than the current model, its going to take no
less than working code. I've seen some literature that leads me to
believe that we could extend the current model to add more operational
modes (and allow for more advanced features in other places). Though
in my priority queue this ranks somewhere around "would be
interesting" and thus is barely a blip on the radar considering all
the other work that needs to be done.

HTH,
Paul Davis

> Scott
>
> On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>>
>>> Why are the validation functions run at replication time?
>>
>> As far as the data store in CouchDB is concerned, replication is
>> just another client doing GETs & PPSTs). Replication uses the
>> same API any user faces.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>>
>>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>>
>>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>>> userCtx.
>>>>>
>>>>
>>>> The problem with this approach is that the validation functions are
>>>> run at replication time as well as at initial update time. This is why
>>>> we need to abstract the http information into a userCtx, because the
>>>> full request object won't be available for replay later (and the
>>>> replicating userCtx is used at rep time, not the original Ctx, so
>>>> replay wouldn't be appropriate anyway.)
>>>>
>>>> Both concerns in this thread (custom http auth & using user creds from
>>>> a db) are best addressed by writing another authentication handler or
>>>> two. I believe there is work underway to do this but I'm not sure the
>>>> current state.
>>>>
>>>> We are very open to patches in the auth handler section of the code.
>>>> Please inquire on dev@ (or drop a patch on
>>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>>> here.
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>> --
>>>> Chris Anderson
>>>> http://jchrisa.net
>>>> http://couch.io
>>>>
>>>
>>
>>
>

Re: how to do http request within server-side _show or _view functions?

[Scott Shumaker <ss...@gmail.com>]

Apparently at the cost of limiting the functionality of dozens of
unrelated features - everything from transactions to validation.

I grow ever more convinced that CouchDB's replication is an albatross.
 It's analogous to RAID-1 - easy to code but pretty limited (and now
very outdated) tech.  It wouldn't be a problem if it were an isolated
feature, but it hamstrings so many others.

Maybe it makes sense for a particular kind of project, but for most
real-world applications you can get better (more scalable, higher
performant, and more reliable) results with some sort of Paxos or
Vector-clock architecture with multiple committers and readers.  I do
hope to see CouchDB move in that direction - and I think it has a good
chance of becoming the web persistent storage engine of choice if it
does.

Scott

On Sun, May 17, 2009 at 1:43 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 17 May 2009, at 01:28, Scott Shumaker wrote:
>
>> Why are the validation functions run at replication time?
>
> As far as the data store in CouchDB is concerned, replication is
> just another client doing GETs & PPSTs). Replication uses the
> same API any user faces.
>
> Cheers
> Jan
> --
>
>
>>
>> On Tue, May 12, 2009 at 5:33 AM, Chris Anderson <jc...@apache.org> wrote:
>>>
>>> On Tue, May 12, 2009 at 1:13 AM, Scott Shumaker <ss...@gmail.com>
>>> wrote:
>>>>
>>>> Not ideal, but it works.  I'd love for validate_doc_update to take
>>>> HTTP headers - especially an additional JSON parameter, like a custom
>>>> userCtx.
>>>>
>>>
>>> The problem with this approach is that the validation functions are
>>> run at replication time as well as at initial update time. This is why
>>> we need to abstract the http information into a userCtx, because the
>>> full request object won't be available for replay later (and the
>>> replicating userCtx is used at rep time, not the original Ctx, so
>>> replay wouldn't be appropriate anyway.)
>>>
>>> Both concerns in this thread (custom http auth & using user creds from
>>> a db) are best addressed by writing another authentication handler or
>>> two. I believe there is work underway to do this but I'm not sure the
>>> current state.
>>>
>>> We are very open to patches in the auth handler section of the code.
>>> Please inquire on dev@ (or drop a patch on
>>> https://issues.apache.org/jira/browse/COUCHDB) if you'd like to help
>>> here.
>>>
>>> Chris
>>>
>>>
>>>
>>> --
>>> Chris Anderson
>>> http://jchrisa.net
>>> http://couch.io
>>>
>>
>
>