Pinning revs

[Brian Candler <B....@pobox.com>]

Have there been any more thoughts about being able to use _rev as a history
mechanism?

I think this just means that certain older _revs can survive compaction, and
ISTM that the simplest way to achieve this would be to have a bit which
marks a particular revision as "pinned" (cannot be discareded).  This would
be very flexible.  For example, you could prune this bit so that you keep
one revision per day for the last week, one revision per month before that,
and so on.  When making an update in a wiki, you could pin the previous
revision only if it's more than 1 hour old, allowing multiple updates within
this window to be coalesced.

I think this would be a very convenient mechanism, and much moreso than
building a document with all the previous versions of interest within the
document itself, or as attachments.

I've even considered introducing artificial conflicts into the database
purely as a way to retain previous revs, but that's pretty messy.

Regards,

Brian.

Re: Pinning revs

[Metin Akat <ak...@gmail.com>]

Absolutely! I'm voting on this one. That would ease my work immensely.

On Sun, Jan 24, 2010 at 12:20 PM, Brian Candler <B....@pobox.com> wrote:
> Have there been any more thoughts about being able to use _rev as a history
> mechanism?
>
> I think this just means that certain older _revs can survive compaction, and
> ISTM that the simplest way to achieve this would be to have a bit which
> marks a particular revision as "pinned" (cannot be discareded).  This would
> be very flexible.  For example, you could prune this bit so that you keep
> one revision per day for the last week, one revision per month before that,
> and so on.  When making an update in a wiki, you could pin the previous
> revision only if it's more than 1 hour old, allowing multiple updates within
> this window to be coalesced.
>
> I think this would be a very convenient mechanism, and much moreso than
> building a document with all the previous versions of interest within the
> document itself, or as attachments.
>
> I've even considered introducing artificial conflicts into the database
> purely as a way to retain previous revs, but that's pretty messy.
>
> Regards,
>
> Brian.
>

Re: Pinning revs

[Paul Davis <pa...@gmail.com>]

On Mon, Jan 25, 2010 at 9:38 AM, Robert Newson <ro...@gmail.com> wrote:
> fwiw, I have the same hinky feeling about this proposal. If
> implemented, it would be the case that revisions are a history
> mechanism under user control, when couch has always, and rightly, said
> that it is not.
>
> when you PUT a new document over an existing one you are implicitly
> removing the document that was previously there. A concurrent read
> operation might see the b+tree before or after that change; either
> answer is consistent with some historical version of the database and
> no locking is required.
>
> If, instead, you really wanted to make a new version (from your
> applications point of view) you should insert a brand new document and
> add a view (or a naming convention) that lets you find the version
> history. A simple idea would be to append the version to the _id.
> (i.e, to 'update' doc1-v1, you would PUT doc1-v2). Purging some or all
> history would then be a sequence of DELETE's up to, and exclusive of,
> the latest version. This approach will work correctly through all
> compaction, replication, multi-master and offline scenarios.
>
> It's not clear if any of that belongs inside couchdb but clearly
> something like it would be useful to a lot of folks. Perhaps it's
> another tool outside of couchdb that, like couchapp, adds some finesse
> over a fundamental concept?
>
> B.
>
>
>
> On Mon, Jan 25, 2010 at 1:08 PM, Robert Dionne
> <di...@dionne-associates.com> wrote:
>> I gave this some more thought over the weekend and don't think it's a good idea. Admittedly I'm not as deep in the code as the core devs but this strikes me as non-trivial to get right. There has also been a lot of effort put into telling folks to not think of _rev as a history mechanism. It seems very doable but as you point out it needs a strategy for retention, which would likely need to be configurable for different scenarios and there would need to be a strategy for replication also, how much history to carry along and so forth.
>>
>> If this is not best done by clients I think something in the server that was entirely orthogonal, .eg. based on some changes notification and using a log or different store, would be better. This would keep the design simpler and enable users to leave it out if not needed.
>>
>> Just my two cents but I'd be -0 on it
>>
>> Regards,
>>
>> Bob
>>
>>
>>
>> On Jan 24, 2010, at 5:20 AM, Brian Candler wrote:
>>
>>> Have there been any more thoughts about being able to use _rev as a history
>>> mechanism?
>>>
>>> I think this just means that certain older _revs can survive compaction, and
>>> ISTM that the simplest way to achieve this would be to have a bit which
>>> marks a particular revision as "pinned" (cannot be discareded).  This would
>>> be very flexible.  For example, you could prune this bit so that you keep
>>> one revision per day for the last week, one revision per month before that,
>>> and so on.  When making an update in a wiki, you could pin the previous
>>> revision only if it's more than 1 hour old, allowing multiple updates within
>>> this window to be coalesced.
>>>
>>> I think this would be a very convenient mechanism, and much moreso than
>>> building a document with all the previous versions of interest within the
>>> document itself, or as attachments.
>>>
>>> I've even considered introducing artificial conflicts into the database
>>> purely as a way to retain previous revs, but that's pretty messy.
>>>
>>> Regards,
>>>
>>> Brian.
>>
>>
>

As Bob and Rob point out, it may seem easy at first blush, but
replication ends up getting fairly complicated. What happens when
you're trying to replicate between two servers that have different
sets of revisions pinned?

HTH,
Paul Davis

Re: Pinning revs

[Brian Candler <B....@pobox.com>]

On Mon, Jan 25, 2010 at 02:38:22PM +0000, Robert Newson wrote:
> when you PUT a new document over an existing one you are implicitly
> removing the document that was previously there. A concurrent read
> operation might see the b+tree before or after that change; either
> answer is consistent with some historical version of the database and
> no locking is required.
> 
> If, instead, you really wanted to make a new version (from your
> applications point of view) you should insert a brand new document and
> add a view (or a naming convention) that lets you find the version
> history.
> 
> A simple idea would be to append the version to the _id.
> (i.e, to 'update' doc1-v1, you would PUT doc1-v2).

That's what I thought of first. Given 1000 revisions of one document, stored
as 1000 separate documents, then (as you say) you can make a view to find
the most recent one.

However, you can't apply a view to a view, so it's then impossible to write
a view which makes use of only the most recent version of a document. It
becomes a bit of a mess.

So I think I need to store all the revisions within a single document.
Options might be:

1. Store all the revisions nested with the JSON document - or store the
prevision revisions as attachments.  Unfortunately, I need to version the
binary attachments too.

2. Store each attachment with a special naming convention, e.g. blob:r1,
blob:r2 etc

3. Store each rev's attachments in a single .zip file attachment.

4. Store each attachment with a name equal to its sha1, and the revisions as
nested JSON each containing an "attachments": member that points to the
sha1's.  Probably the cleanest and also saves duplicating identical content,
but still something of a PITA.

I guess that, as you say, it could be layered on-top of couchdb as some sort
of middleware, or else the client would have to take responsibility for
doing the versioning properly.

(An _update handler could update the JSON part of a multi-rev document, but
I don't think it can do clever stuff with attachments)

Regards,

Brian.

Re: Pinning revs

[Chris Anderson <jc...@apache.org>]

On Mon, Jan 25, 2010 at 7:24 AM, Joscha Feth <jo...@feth.com> wrote:
> Robert Newson wrote:
>
>> It's not clear if any of that belongs inside couchdb but clearly
>> something like it would be useful to a lot of folks. Perhaps it's
>> another tool outside of couchdb that, like couchapp, adds some finesse
>> over a fundamental concept?
>
> But isn't there the chance that any external program might miss some
> revisions if old revisions are discarded prior to being picked up by
> any external process fetching them?

It's complications like this that point to a document (or attachment)
per revision model for wiki-like storage.

The more I think about it the more I realize Damien is right that for
versioned storage you should attach the old version as a text/json
attachment.

This way you don't have to deal with the mass of old revisions. Also,
each attachment update is atomic with the new data. The best way to
compress something like this would be to run it on a
content-addressable filesystem.

Chris

>
> Let's say we have a scenario like this with a program fetching
> revisions based on a cronjob which runs in a predefined interval:
>
> A has rev-1
> fetching revisions
> Update on A. Revison gets bumped to rev-2
> Compaction run on database
> Update on A. Revison gets bumped to rev-3
> fetching revisions <-- no way to fetch rev-2 as it got deleted already.
>
> so for using an external program to do the revision work, there is no
> way to just do this at a predefined interval, except if compactions can
> be suppressed until it runs again and as far as I understand the
> current design might not guarantee this?!
>
> So the only other option would be a program which listens on _changes
> and uses the ?since parameter - what happens to the scenario above
> then? Are events on _changes preserved even over compaction?
>
>
> regards,
> Joscha
>
>
> --
>
>
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: Pinning revs

[Joscha Feth <jo...@feth.com>]

Robert Newson wrote:

> It's not clear if any of that belongs inside couchdb but clearly
> something like it would be useful to a lot of folks. Perhaps it's
> another tool outside of couchdb that, like couchapp, adds some finesse
> over a fundamental concept?

But isn't there the chance that any external program might miss some
revisions if old revisions are discarded prior to being picked up by
any external process fetching them?

Let's say we have a scenario like this with a program fetching
revisions based on a cronjob which runs in a predefined interval:

A has rev-1
fetching revisions
Update on A. Revison gets bumped to rev-2
Compaction run on database
Update on A. Revison gets bumped to rev-3
fetching revisions <-- no way to fetch rev-2 as it got deleted already.

so for using an external program to do the revision work, there is no
way to just do this at a predefined interval, except if compactions can
be suppressed until it runs again and as far as I understand the
current design might not guarantee this?!

So the only other option would be a program which listens on _changes
and uses the ?since parameter - what happens to the scenario above
then? Are events on _changes preserved even over compaction?


regards,
Joscha


-- 

Re: Pinning revs

[Robert Newson <ro...@gmail.com>]

fwiw, I have the same hinky feeling about this proposal. If
implemented, it would be the case that revisions are a history
mechanism under user control, when couch has always, and rightly, said
that it is not.

when you PUT a new document over an existing one you are implicitly
removing the document that was previously there. A concurrent read
operation might see the b+tree before or after that change; either
answer is consistent with some historical version of the database and
no locking is required.

If, instead, you really wanted to make a new version (from your
applications point of view) you should insert a brand new document and
add a view (or a naming convention) that lets you find the version
history. A simple idea would be to append the version to the _id.
(i.e, to 'update' doc1-v1, you would PUT doc1-v2). Purging some or all
history would then be a sequence of DELETE's up to, and exclusive of,
the latest version. This approach will work correctly through all
compaction, replication, multi-master and offline scenarios.

It's not clear if any of that belongs inside couchdb but clearly
something like it would be useful to a lot of folks. Perhaps it's
another tool outside of couchdb that, like couchapp, adds some finesse
over a fundamental concept?

B.



On Mon, Jan 25, 2010 at 1:08 PM, Robert Dionne
<di...@dionne-associates.com> wrote:
> I gave this some more thought over the weekend and don't think it's a good idea. Admittedly I'm not as deep in the code as the core devs but this strikes me as non-trivial to get right. There has also been a lot of effort put into telling folks to not think of _rev as a history mechanism. It seems very doable but as you point out it needs a strategy for retention, which would likely need to be configurable for different scenarios and there would need to be a strategy for replication also, how much history to carry along and so forth.
>
> If this is not best done by clients I think something in the server that was entirely orthogonal, .eg. based on some changes notification and using a log or different store, would be better. This would keep the design simpler and enable users to leave it out if not needed.
>
> Just my two cents but I'd be -0 on it
>
> Regards,
>
> Bob
>
>
>
> On Jan 24, 2010, at 5:20 AM, Brian Candler wrote:
>
>> Have there been any more thoughts about being able to use _rev as a history
>> mechanism?
>>
>> I think this just means that certain older _revs can survive compaction, and
>> ISTM that the simplest way to achieve this would be to have a bit which
>> marks a particular revision as "pinned" (cannot be discareded).  This would
>> be very flexible.  For example, you could prune this bit so that you keep
>> one revision per day for the last week, one revision per month before that,
>> and so on.  When making an update in a wiki, you could pin the previous
>> revision only if it's more than 1 hour old, allowing multiple updates within
>> this window to be coalesced.
>>
>> I think this would be a very convenient mechanism, and much moreso than
>> building a document with all the previous versions of interest within the
>> document itself, or as attachments.
>>
>> I've even considered introducing artificial conflicts into the database
>> purely as a way to retain previous revs, but that's pretty messy.
>>
>> Regards,
>>
>> Brian.
>
>

Re: Pinning revs

[Robert Dionne <di...@dionne-associates.com>]

I gave this some more thought over the weekend and don't think it's a good idea. Admittedly I'm not as deep in the code as the core devs but this strikes me as non-trivial to get right. There has also been a lot of effort put into telling folks to not think of _rev as a history mechanism. It seems very doable but as you point out it needs a strategy for retention, which would likely need to be configurable for different scenarios and there would need to be a strategy for replication also, how much history to carry along and so forth. 

If this is not best done by clients I think something in the server that was entirely orthogonal, .eg. based on some changes notification and using a log or different store, would be better. This would keep the design simpler and enable users to leave it out if not needed.

Just my two cents but I'd be -0 on it

Regards,

Bob



On Jan 24, 2010, at 5:20 AM, Brian Candler wrote:

> Have there been any more thoughts about being able to use _rev as a history
> mechanism?
> 
> I think this just means that certain older _revs can survive compaction, and
> ISTM that the simplest way to achieve this would be to have a bit which
> marks a particular revision as "pinned" (cannot be discareded).  This would
> be very flexible.  For example, you could prune this bit so that you keep
> one revision per day for the last week, one revision per month before that,
> and so on.  When making an update in a wiki, you could pin the previous
> revision only if it's more than 1 hour old, allowing multiple updates within
> this window to be coalesced.
> 
> I think this would be a very convenient mechanism, and much moreso than
> building a document with all the previous versions of interest within the
> document itself, or as attachments.
> 
> I've even considered introducing artificial conflicts into the database
> purely as a way to retain previous revs, but that's pretty messy.
> 
> Regards,
> 
> Brian.

Re: Pinning revs

[Joscha Feth <jo...@feth.com>]

Brian Candler wrote:

> Have there been any more thoughts about being able to use _rev as a
> history mechanism?

+1 from here - this would make the revision scenario for my current
project incredibly easy!

regards,
Joscha

--