Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jason Huggins <ja...@jrandolph.com>]

Hello all!

I would like some meta-data to get inserted into documents when my
users create them. At a minimum, something like Created and
LastModified datetime stamps would be nice. Currently, the users have
to manually add these fields when they POST or PUT. Ideally, there is
something I can do on the server side to "automagically" add those
fields for them.

I think the easiest thing to do is to create a DBNotificationUpdate
script that looks for new documents and updates them with the datetime
data as a document revision.

Is there a better way to handle this? I think I want something like
what happens now with the creation of the "_id" field... If "_id" is
missing, CouchDB will add it to the document on the user's behalf at
document creation time.

Thanks, and happy couching!

-jason
http://saucelabs.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Antony Blakey <an...@gmail.com>]

On 04/12/2008, at 9:31 AM, Chris Anderson wrote:

> On Wed, Dec 3, 2008 at 2:50 PM, Antony Blakey  
> <an...@gmail.com> wrote:
>>
>> On 04/12/2008, at 9:07 AM, Chris Anderson wrote:
>>
>>> It should just be clear that timestamps are the application's
>>> business, not the database's.
>>
>> But it's possible for Couch to be the application, especially if  
>> you use
>> your apps-in-the-db approach. I'm not sure I see any fundamental  
>> difference
>> between a validation function in a design document and some  
>> javascript in an
>> attachment, of even a function inan _external handler.
>>
>
> The problem with writing from the validation functions is that you are
> supposed to be able to trust them. Eg, if you look at the function,
> you can know what contract holds on the data that passes through it. A
> timestamp on new docs doesn't pass this test. There's no way to know
> whether the timestamp was added by a different function. With proper
> pass/fail validations, you know that running the same validations at
> replication time will give the same pass/fail result as at original
> write time.

IIUC, replication conflicts in a P2P configuration allow for a  
situation where even a validator that is purely funtional wrt the  
database state can generate different results upon replication. Or  
maybe I'm missing something?

Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

A reasonable man adapts himself to suit his environment. An  
unreasonable man persists in attempting to adapt his environment to  
suit himself. Therefore, all progress depends on the unreasonable man.
   -- George Bernard Shaw

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Chris Anderson <jc...@apache.org>]

On Wed, Dec 3, 2008 at 2:50 PM, Antony Blakey <an...@gmail.com> wrote:
>
> On 04/12/2008, at 9:07 AM, Chris Anderson wrote:
>
>> It should just be clear that timestamps are the application's
>> business, not the database's.
>
> But it's possible for Couch to be the application, especially if you use
> your apps-in-the-db approach. I'm not sure I see any fundamental difference
> between a validation function in a design document and some javascript in an
> attachment, of even a function inan _external handler.
>

The problem with writing from the validation functions is that you are
supposed to be able to trust them. Eg, if you look at the function,
you can know what contract holds on the data that passes through it. A
timestamp on new docs doesn't pass this test. There's no way to know
whether the timestamp was added by a different function. With proper
pass/fail validations, you know that running the same validations at
replication time will give the same pass/fail result as at original
write time.

It's nice that _external will be in trunk and available for this sort
of thing. (Application in the db.)

-- 
Chris Anderson
http://jchris.mfdz.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Antony Blakey <an...@gmail.com>]

On 04/12/2008, at 9:07 AM, Chris Anderson wrote:

> It should just be clear that timestamps are the application's
> business, not the database's.

But it's possible for Couch to be the application, especially if you  
use your apps-in-the-db approach. I'm not sure I see any fundamental  
difference between a validation function in a design document and some  
javascript in an attachment, of even a function inan _external handler.

Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

Hi, I'd like to do $THING. I know that $SOLUTION_A and $SOLUTION_B  
will do it very easily and for a very reasonable price, but I don't  
want to use $SOLUTION_A or $SOLUTION_B because $VAGUE_REASON and  
$CONTRADICTORY_REASON. Instead, I'd like your under-informed ideas on  
how to achieve my $POORLY_CONCEIVED_AMBITIONS using Linux, duct tape,  
an iPod, and hours and hours of my precious time.
   -- Slashdot response to an enquiry

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Chris Anderson <jc...@apache.org>]

Ha, this thread seems to have split on both user and dev.

On Wed, Dec 3, 2008 at 1:00 AM, Jason Huggins <ja...@jrandolph.com> wrote:
> On Tue, Dec 2, 2008 at 8:42 PM, Chris Anderson <jc...@apache.org> wrote:
>> I think I've had a change of heart. Damien argues that people would
>> use validation-writes to generate timestamps. But the timestamps can
>> be wrong/faked (on remote nodes) and there's no way to validate (on
>> replication) that they are correct. So putting stamp-capability into
>> the validation functions would setup false expectations. Apparently it
>> caused all sorts of trouble in Notes.
>
> Would this validate, then, my idea for a 2nd "meta-data db" that only
> the server had POST/PUT/DELETE rights to, but users could GET from?
>

Its ok to have timestamps, it would just set the wrong expectation for
CouchDB to set them inside the validation function. Because there is
no way to validate that a timestamp is correct, especially when
replicating from a remote source, setting them inside the database
logic itself is wrong. However, using something like _external (or to
otherwise reprocess with a batch) to set them or other server
variables would be fine.

It should just be clear that timestamps are the application's
business, not the database's.

-- 
Chris Anderson
http://jchris.mfdz.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jason Huggins <ja...@jrandolph.com>]

On Tue, Dec 2, 2008 at 8:42 PM, Chris Anderson <jc...@apache.org> wrote:
> I think I've had a change of heart. Damien argues that people would
> use validation-writes to generate timestamps. But the timestamps can
> be wrong/faked (on remote nodes) and there's no way to validate (on
> replication) that they are correct. So putting stamp-capability into
> the validation functions would setup false expectations. Apparently it
> caused all sorts of trouble in Notes.

Would this validate, then, my idea for a 2nd "meta-data db" that only
the server had POST/PUT/DELETE rights to, but users could GET from?

-j.

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Chris Anderson <jc...@apache.org>]

On Tue, Dec 2, 2008 at 11:10 PM, Jedediah Smith
<je...@silencegreys.com> wrote:
>
> How about "live
> views"? e.g.
>
> /db/_view/email/by_date?follow=true&after_seq=300
>
> But then you need a way to represent removed rows.. seems simple enough:
>
> {"id":"abc", "key":"123", "deleted":true}
>
>

This is essentially how replication via views will work. Replication
views have slightly different semantics (they aren't stored on the
server, for one thing -- it's up to the client to maintain the
replication state.)

As a result their function signature is different (they don't emit as
they aren't sorted by key, just seq #). The client supplies a
replication function, which the server runs across docs in seq order,
replicating docs that match the function. This is looking very
powerful.

Chris

-- 
Chris Anderson
http://jchris.mfdz.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jedediah Smith <je...@silencegreys.com>]

Chris Anderson wrote:
> On Tue, Dec 2, 2008 at 12:01 AM, Jedediah Smith
> <je...@silencegreys.com> wrote:
>> If validations could write, they could serve as a complete data access
>> layer. They could handle meta data generation, merging, encapsulation (aka
>> implementation hiding/abstraction/conversion) and essentially solve the
>> partial update problem.
> 
> I think I've had a change of heart. Damien argues that people would
> use validation-writes to generate timestamps. But the timestamps can
> be wrong/faked (on remote nodes) and there's no way to validate (on
> replication) that they are correct. So putting stamp-capability into
> the validation functions would setup false expectations. Apparently it
> caused all sorts of trouble in Notes.
> 
> This is an instance of a broader distributed computing axiom: no global clock.

If CouchDB can't add timestamps then the client has to do it and the 
client is sometimes a web browser. Who's more likely to give you bad data?

If someone needs timestamps in a multi-master db cluster, then they need 
timestamps in a multi-master db cluster. Might as well throw them a 
bone. Anybody in charge of building such a thing is probably somebody 
who doesn't need their hand held anyway. Notes was a user app.. totally 
different situation.

Oddly, I had to deal with this very problem today at work. Some time 
deltas were way off and we discovered that changing the clock on the db 
server affected it. But our problem was that a service was running on 
the wrong server.

>> If merges could be handled by script functions, two-way replication could be
>> completely autonomous. Plus, sloppy developers would be more likely to
>> actually handle revision conflicts instead of just ignoring them.
> 
> It might be best to save this action for the end of replication, as
> replication could apply more than one replicated rev per document. It
> would be nice to have a place in design docs for replication conflict
> revision, but I don't think validation is that place.
> 
>> Why not just provide hooks to every part of CouchDB?
> 
> I think the important thing to do is design them in such a way as to
> give developers the flexibility they need while maintaining the right
> expectations about what a distributed database can offer.
> 
> That said, there is a lot of work to add really cool hooks into
> CouchDB. One of the most exciting prospects is Comet requests, where
> you could do a POST to /db/_since_seq?seq=300 and wait for a response
> until the next change after the last change that you knew about
> happened.

Oh, absolutely.. have many handlers each with very specific semantic 
requirements. They'll need a way to share code though.

Comet is cool and it will be needed in a world of short-stack AJAX apps. 
But a feed of all updates will be too much data in some cases. How about 
"live views"? e.g.

/db/_view/email/by_date?follow=true&after_seq=300

But then you need a way to represent removed rows.. seems simple enough:

{"id":"abc", "key":"123", "deleted":true}


Maybe we should move this to couchdb-dev?

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jedediah Smith <je...@silencegreys.com>]

Chris Anderson wrote:
> On Tue, Dec 2, 2008 at 12:01 AM, Jedediah Smith
> <je...@silencegreys.com> wrote:
>> If validations could write, they could serve as a complete data access
>> layer. They could handle meta data generation, merging, encapsulation (aka
>> implementation hiding/abstraction/conversion) and essentially solve the
>> partial update problem.
> 
> I think I've had a change of heart. Damien argues that people would
> use validation-writes to generate timestamps. But the timestamps can
> be wrong/faked (on remote nodes) and there's no way to validate (on
> replication) that they are correct. So putting stamp-capability into
> the validation functions would setup false expectations. Apparently it
> caused all sorts of trouble in Notes.
> 
> This is an instance of a broader distributed computing axiom: no global clock.

If CouchDB can't add timestamps then the client has to do it and the 
client is sometimes a web browser. Who's more likely to give you bad data?

If someone needs timestamps in a multi-master db cluster, then they need 
timestamps in a multi-master db cluster. Might as well throw them a 
bone. Anybody in charge of building such a thing is probably somebody 
who doesn't need their hand held anyway. Notes was a user app.. totally 
different situation.

Oddly, I had to deal with this very problem today at work. Some time 
deltas were way off and we discovered that changing the clock on the db 
server affected it. But our problem was that a service was running on 
the wrong server.

>> If merges could be handled by script functions, two-way replication could be
>> completely autonomous. Plus, sloppy developers would be more likely to
>> actually handle revision conflicts instead of just ignoring them.
> 
> It might be best to save this action for the end of replication, as
> replication could apply more than one replicated rev per document. It
> would be nice to have a place in design docs for replication conflict
> revision, but I don't think validation is that place.
> 
>> Why not just provide hooks to every part of CouchDB?
> 
> I think the important thing to do is design them in such a way as to
> give developers the flexibility they need while maintaining the right
> expectations about what a distributed database can offer.
> 
> That said, there is a lot of work to add really cool hooks into
> CouchDB. One of the most exciting prospects is Comet requests, where
> you could do a POST to /db/_since_seq?seq=300 and wait for a response
> until the next change after the last change that you knew about
> happened.

Oh, absolutely.. have many handlers each with very specific semantic 
requirements. They'll need a way to share code though.

Comet is cool and it will be needed in a world of short-stack AJAX apps. 
But a feed of all updates will be too much data in some cases. How about 
"live views"? e.g.

/db/_view/email/by_date?follow=true&after_seq=300

But then you need a way to represent removed rows.. seems simple enough:

{"id":"abc", "key":"123", "deleted":true}


Maybe we should move this to couchdb-dev?

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Chris Anderson <jc...@apache.org>]

On Tue, Dec 2, 2008 at 12:01 AM, Jedediah Smith
<je...@silencegreys.com> wrote:
>
> If validations could write, they could serve as a complete data access
> layer. They could handle meta data generation, merging, encapsulation (aka
> implementation hiding/abstraction/conversion) and essentially solve the
> partial update problem.

I think I've had a change of heart. Damien argues that people would
use validation-writes to generate timestamps. But the timestamps can
be wrong/faked (on remote nodes) and there's no way to validate (on
replication) that they are correct. So putting stamp-capability into
the validation functions would setup false expectations. Apparently it
caused all sorts of trouble in Notes.

This is an instance of a broader distributed computing axiom: no global clock.

>
> If merges could be handled by script functions, two-way replication could be
> completely autonomous. Plus, sloppy developers would be more likely to
> actually handle revision conflicts instead of just ignoring them.

It might be best to save this action for the end of replication, as
replication could apply more than one replicated rev per document. It
would be nice to have a place in design docs for replication conflict
revision, but I don't think validation is that place.

>
> Why not just provide hooks to every part of CouchDB?

I think the important thing to do is design them in such a way as to
give developers the flexibility they need while maintaining the right
expectations about what a distributed database can offer.

That said, there is a lot of work to add really cool hooks into
CouchDB. One of the most exciting prospects is Comet requests, where
you could do a POST to /db/_since_seq?seq=300 and wait for a response
until the next change after the last change that you knew about
happened.

-- 
Chris Anderson
http://jchris.mfdz.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jedediah Smith <je...@silencegreys.com>]

If validations could write, they could serve as a complete data access 
layer. They could handle meta data generation, merging, encapsulation 
(aka implementation hiding/abstraction/conversion) and essentially solve 
the partial update problem.

If merges could be handled by script functions, two-way replication 
could be completely autonomous. Plus, sloppy developers would be more 
likely to actually handle revision conflicts instead of just ignoring them.

 From what I understand of the current architecture, it's most of the 
way there. There's already a script engine, a way to store scripts 
associated with an event, a protocol for invoking them and for passing 
data back and forth. Why not just provide hooks to every part of CouchDB?

Chris Anderson wrote:
> On Mon, Dec 1, 2008 at 7:07 PM, Jason Huggins <ja...@jrandolph.com> wrote:
>> On Mon, Dec 1, 2008 at 6:56 PM, Jedediah Smith
>> <je...@silencegreys.com> wrote:
>>> The new validation functions might be able to do this for you if they are
>>> allowed to modify the document. Whether or not they are was asked on the dev
>>> list but not answered. I hope they are as that would be endlessly useful. It
>>> would be *really* neat if a function could also handle merging for revision
>>> conflicts.
> 
> I'm not sure about changing documents -- I think it would be simple to
> add if the demand is strong.
> 
>> Are validation functions in trunk or a released version?... or still
>> on the drawing board? (I suppose I should switch to the dev list to
>> ask that question to a more targeted audience.) :-)
>>
> 
> I wrote up my dive through the security_validation tests here:
> 
> http://jchris.mfdz.com/code/2008/12/couchdb_edge__security_and_vali
> 
> Validations are in trunk, but you'd have to do some work to use them
> in an application.
> 
> 

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Chris Anderson <jc...@apache.org>]

On Mon, Dec 1, 2008 at 7:07 PM, Jason Huggins <ja...@jrandolph.com> wrote:
> On Mon, Dec 1, 2008 at 6:56 PM, Jedediah Smith
> <je...@silencegreys.com> wrote:
>> The new validation functions might be able to do this for you if they are
>> allowed to modify the document. Whether or not they are was asked on the dev
>> list but not answered. I hope they are as that would be endlessly useful. It
>> would be *really* neat if a function could also handle merging for revision
>> conflicts.

I'm not sure about changing documents -- I think it would be simple to
add if the demand is strong.

> Are validation functions in trunk or a released version?... or still
> on the drawing board? (I suppose I should switch to the dev list to
> ask that question to a more targeted audience.) :-)
>

I wrote up my dive through the security_validation tests here:

http://jchris.mfdz.com/code/2008/12/couchdb_edge__security_and_vali

Validations are in trunk, but you'd have to do some work to use them
in an application.


-- 
Chris Anderson
http://jchris.mfdz.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jason Huggins <ja...@jrandolph.com>]

On Mon, Dec 1, 2008 at 6:56 PM, Jedediah Smith
<je...@silencegreys.com> wrote:
> The new validation functions might be able to do this for you if they are
> allowed to modify the document. Whether or not they are was asked on the dev
> list but not answered. I hope they are as that would be endlessly useful. It
> would be *really* neat if a function could also handle merging for revision
> conflicts.

Are validation functions in trunk or a released version?... or still
on the drawing board? (I suppose I should switch to the dev list to
ask that question to a more targeted audience.) :-)

I have another idea for tracking timestamps and other meta-data.  I
think I'll create a meta-data database for each "real" database.
Since I'd like to give my users free reign to edit their own documents
in the real database, but *not* be able to edit createdAt and
lastModified timestamps. So, with a meta-data db, I can set it up so
only my server has write permission to that meta-db, but the user can
read from it.

The downside is that combining the data and meta-data into a unified
view of the document would have to happen on the client side, not on
the couchdb side, since (as I understand it), couchdb can't do
cross-database map/reducing views.

- jason huggins
  http://saucelabs.com

Re: Auto-adding additional fields on PUT/POST? (datetime stamps)

[Jedediah Smith <je...@silencegreys.com>]

The new validation functions might be able to do this for you if they 
are allowed to modify the document. Whether or not they are was asked on 
the dev list but not answered. I hope they are as that would be 
endlessly useful. It would be *really* neat if a function could also 
handle merging for revision conflicts.

Jason Huggins wrote:
> Hello all!
> 
> I would like some meta-data to get inserted into documents when my
> users create them. At a minimum, something like Created and
> LastModified datetime stamps would be nice. Currently, the users have
> to manually add these fields when they POST or PUT. Ideally, there is
> something I can do on the server side to "automagically" add those
> fields for them.
> 
> I think the easiest thing to do is to create a DBNotificationUpdate
> script that looks for new documents and updates them with the datetime
> data as a document revision.
> 
> Is there a better way to handle this? I think I want something like
> what happens now with the creation of the "_id" field... If "_id" is
> missing, CouchDB will add it to the document on the user's behalf at
> document creation time.
> 
> Thanks, and happy couching!
> 
> -jason
> http://saucelabs.com
>