You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Adam Kocoloski <ko...@apache.org> on 2010/10/26 20:59:50 UTC
format of database sequence
Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
"1234-Base64Data"
The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
Adam
Re: format of database sequence
Posted by Jan Lehnardt <ja...@apache.org>.
On 26 Oct 2010, at 20:59, Adam Kocoloski wrote:
> Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
>
> "1234-Base64Data"
>
> The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
>
> 1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
>
> 2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
>
> My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
The next major release is 2.0. +1 on getting that fixed.
Cheers
Jan
--
Re: format of database sequence
Posted by Dirkjan Ochtman <di...@ochtman.nl>.
On Tue, Oct 26, 2010 at 22:33, Adam Kocoloski <ko...@apache.org> wrote:
> rnewson advocated doing away with the prefix altogether and making it completely opaque. BigCouch used to work this way; we added the prefix because we found it convenient to get a rough idea of the number of updates the DB had seen, but it's hardly necessary. If we did make it opaque we'd certainly need to fix the replicator so that it saves checkpoints regardless of sorting order.
It definitely seems nice if we can still get a rough idea of the
number of updates from the new sequence identifier.
Cheers,
Dirkjan
Re: format of database sequence
Posted by Adam Kocoloski <ko...@apache.org>.
On Oct 26, 2010, at 3:23 PM, Paul Davis wrote:
> On Tue, Oct 26, 2010 at 3:06 PM, Randall Leeds <ra...@gmail.com> wrote:
>> I don't see any way to avoid this and we've been talking about it for a while.
>> +1
>>
>> On Tue, Oct 26, 2010 at 11:59, Adam Kocoloski <ko...@apache.org> wrote:
>>> Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
>>>
>>> "1234-Base64Data"
>>>
>>> The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
>>>
>>> 1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
>>>
>>> 2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
>>>
>>> My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
>>>
>>> Adam
>>>
>>>
>>
>
> Is there a possibility to retain a guarantee of triangle inequality
> for update sequences? In my experience people are mostly using them to
> detect an ordering. It seems like if we could give a loose ordering,
> that'd be better than just an opaque type. Though, I'm also not sure
> how that'd work in relation to string comparison operators in most
> languages.
>
> I'm +1 on the switch, I'm just wondering if we can do it without
> making them completely opaque.
>
> Paul Davis
Logging some IRC discussion. In my opinion the easiest way to get the cluster sequence to sort nicely is to format it as a JSON array instead of a string
[1234, "Base64Data"]
Now, to be clear, sorting of the sequences is not really something that should be encouraged, and in fact should probably should be actively discouraged. Moreover, a proper sorting of two sequences would need to inspect the Base64Data (a compressed vector clock) of each sequence and do the usual vclock comparison logic. Sorting on the prefixes is only a shorthand that works when the same shards are used to generate both sequences.
rnewson advocated doing away with the prefix altogether and making it completely opaque. BigCouch used to work this way; we added the prefix because we found it convenient to get a rough idea of the number of updates the DB had seen, but it's hardly necessary. If we did make it opaque we'd certainly need to fix the replicator so that it saves checkpoints regardless of sorting order.
Adam
Re: format of database sequence
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Oct 26, 2010 at 3:23 PM, Paul Davis <pa...@gmail.com> wrote:
> On Tue, Oct 26, 2010 at 3:06 PM, Randall Leeds <ra...@gmail.com> wrote:
>> I don't see any way to avoid this and we've been talking about it for a while.
>> +1
>>
>> On Tue, Oct 26, 2010 at 11:59, Adam Kocoloski <ko...@apache.org> wrote:
>>> Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
>>>
>>> "1234-Base64Data"
>>>
>>> The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
>>>
>>> 1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
>>>
>>> 2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
>>>
>>> My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
>>>
>>> Adam
>>>
>>>
>>
>
> Is there a possibility to retain a guarantee of triangle inequality
> for update sequences? In my experience people are mostly using them to
> detect an ordering. It seems like if we could give a loose ordering,
> that'd be better than just an opaque type. Though, I'm also not sure
> how that'd work in relation to string comparison operators in most
> languages.
>
> I'm +1 on the switch, I'm just wondering if we can do it without
> making them completely opaque.
>
> Paul Davis
>
I've since been informed that not only do I have no idea what I'm
talking about, that its also not possible. So I stand at +1 for an
opaque string. To make up for the noise I'll include a link:
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/6a16efa375532182
Paul Davis
Re: format of database sequence
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Oct 26, 2010 at 3:06 PM, Randall Leeds <ra...@gmail.com> wrote:
> I don't see any way to avoid this and we've been talking about it for a while.
> +1
>
> On Tue, Oct 26, 2010 at 11:59, Adam Kocoloski <ko...@apache.org> wrote:
>> Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
>>
>> "1234-Base64Data"
>>
>> The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
>>
>> 1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
>>
>> 2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
>>
>> My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
>>
>> Adam
>>
>>
>
Is there a possibility to retain a guarantee of triangle inequality
for update sequences? In my experience people are mostly using them to
detect an ordering. It seems like if we could give a loose ordering,
that'd be better than just an opaque type. Though, I'm also not sure
how that'd work in relation to string comparison operators in most
languages.
I'm +1 on the switch, I'm just wondering if we can do it without
making them completely opaque.
Paul Davis
Re: format of database sequence
Posted by Randall Leeds <ra...@gmail.com>.
I don't see any way to avoid this and we've been talking about it for a while.
+1
On Tue, Oct 26, 2010 at 11:59, Adam Kocoloski <ko...@apache.org> wrote:
> Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
>
> "1234-Base64Data"
>
> The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
>
> 1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
>
> 2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
>
> My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
>
> Adam
>
>
Re: format of database sequence
Posted by Klaus Trainer <kl...@web.de>.
+1
On Tue, 2010-10-26 at 14:59 -0400, Adam Kocoloski wrote:
> Hi all, I've been meaning to bring this up for a while. CouchDB uses integer sequence numbers in the _changes feed and update_seq values, but I don't see any sensible way to preserve that interface in BigCouch. The database sequence in BigCouch needs to combine the sequences of several database shards; currently it's a string formatted like
>
> "1234-Base64Data"
>
> The first piece is the sum of the shard sequence numbers and is not actually used by BigCouch. The second piece is the actual data about the state of the cluster. This format causes a couple of issues:
>
> 1) the replicator occasionally sorts sequence numbers and when it does so, it sorts the BigCouch ones lexicographically and concludes that e.g. "99-..." is the only checkpoint it will ever need to store.
>
> 2) client libraries might not treat the sequence as an opaque data type and may break when operating against a BigCouch.
>
> My personal preference would be to change the format of the Apache CouchDB sequence to a string at the next major release. Thoughts?
>
> Adam
>