You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Sunny Hirai <th...@gmail.com> on 2009/01/12 00:30:09 UTC
Can I guarantee uniqueness in a field without using _id?
First of all, I'd like to say that CouchDB is an amazing project. I've been
following it for a year and have recently made it a first class citizen of
my framework.
Anyways, looked for this answer but I couldn't find it.
Let's say I want to create a database that contains the pages in a wiki.
Each document looks something like this:
{
name: 'about_us',
body: 'This is the about us page'
}
There would be a view on the database where the key is 'name'.
The problem is this:
User A goes to create the 'aboutus' page. We want it to be unique so we
check if it is in the view. Since it is the first 'about_us' page, we can't
find it. So we are about to do the insert but haven't started yet.
In the meantime, User B goes to create the 'aboutus' page. We check the view
and it also isn't there yet because User A has not completed its insert. We
start to do the insert for User B.
In this case, both User A and User B have begun the insert process.
At the end of the inserts, we will have two documents with the same name
'aboutus'. Can this be prevented?
I know I can put the 'name' in the '_id' field but I'd like to avoid it
because I'm abstracting CouchDB and the abstraction does not allow for it.
Anybody have any solutions other than making 'name' the '_id' field?
-Sunny
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Antony Blakey <an...@gmail.com>.
On 13/01/2009, at 12:24 PM, Paul Davis wrote:
> Also, I just realized that an easy way get close to what you want is
> to use MD5's of all the stuff you want to be unique. Then your id's
> remain reasonable unique etc.
Which is a form of global identity, modulo the probabilistic nature of
hash collisions, rather than state.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
75% of statistics are made up on the spot.
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Paul Davis <pa...@gmail.com>.
Sunny,
Also, I just realized that an easy way get close to what you want is
to use MD5's of all the stuff you want to be unique. Then your id's
remain reasonable unique etc.
Paul
On Mon, Jan 12, 2009 at 6:05 PM, Sunny Hirai <th...@gmail.com> wrote:
>>
>> You have just identified the problem and solution that CouchDB solves
>> with using a traditional RDBMS in a distributed environment. Check out
>> the discussion on the CAP hypothesis (Others call it a theorem, I'm
>> not convinced, but the basic idea is there).
>>
>> Or more succinctly, guaranteed uniqueness requires global state,
>> global state does not scale.
>>
>> HTH,
>> Paul Davis
>>
>
> Yeah, that's what I thought but I was hoping for something in between.
>
> That is, all I want to do is guarantee uniqueness on one or a set of fields.
> It seems to me that it would not require locks on reads and most of eventual
> consistency would be okay except for when dealing with an INSERT or an
> UPDATE that changes the unique field. All other cases could work just like
> before. It would only require a special locked lookup on the unique field or
> field set which could be optimized.
>
> I know that CouchDB is not a solution to all problems (nor do I think it
> should be) but it seems that this would be common for most web applications
> which is a major CouchDB target. For example, almost any website needs
> unique user_names. They also need unique names for pages in a wiki. Blogs
> that use named pages would also have this problem.
>
> In other words, this seems like an issue that happens so often in the web
> computing world that I wonder if it should be supported in CouchDB.
>
> Of course, the argument is to push that global state into the application
> layer and store it elsewhere; however, the question is whether that
> complexity would better service the developer if it was (a) in CouchDB or
> (b) in the application itself.
>
> Since it is a common case (as opposed to an edge case), perhaps there could
> be some discussion of putting it into CouchDB.
>
> Seriously though, would this not affect 100% of websites with a user base?
> One solution, I suppose, is to store the user data in an SQL database, but I
> prefer having as much as possible in CouchDB due to its flexible schemas.
>
> Sunny
>
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Chris Anderson <jc...@gmail.com>.
On Mon, Jan 12, 2009 at 5:12 PM, James Marca
<jm...@translab.its.uci.edu> wrote:
> It always strikes me
> as a bit of a hack to overload the _id just to use its uniqueness
> features. But given that one *can* get unique fields, I'd guess there
> are other things for the developers to solve first.
Supporting multiple unique keys would not lead us closer to the goal
of getting CouchDB to provide its current HTTP API, even when run
across a partitioned cluster of very many nodes.
The same story that applies about global state and uniqueness outside
of CouchDB also applies inside. The magic of hash functions allow us
to provide a single uniqueness constraint. If you need multiple
distinct unique keys, you'll need to track them as separate documents,
and deal with race conditions in your application. (There may be a
hack to get the _bulk_docs API to help you a little here, but it might
be that _bulk_docs transactions will have go away for the sake of
scalability.) Eventually consistent cluster transactions will be a
good challenge for client libraries.
I can understand the value you find in using multiple keys, but the
best way to get your application into Couch will be design it for
standalone documents. I'm glad it looks like you can get by without
multiple keys, there's a lot you can do with application-controlled
ids.
--
Chris Anderson
http://jchris.mfdz.com
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Paul Davis <pa...@gmail.com>.
Brief aside:
1. The first sentence of the wiki database article [1]:
A database is a structured collection of records or data that is
stored in a computer system.
2. The first sentence of http://couchdb.apache.org:
Apache CouchDB is a distributed, fault-tolerant and schema-free
document-oriented database accessible via a RESTful HTTP/JSON API.
3. The only time "document" occurs in the database article is:
See "Locking tables and databases" (section in some documentation /
explanation from IBM) for more detail.
That is all.
[1] http://en.wikipedia.org/wiki/Database
On Mon, Jan 12, 2009 at 11:21 PM, Chris Anderson <jc...@gmail.com> wrote:
> On Mon, Jan 12, 2009 at 7:51 PM, Jason Huggins <ja...@jrandolph.com> wrote:
>
>> "Database? Don't you mean 'docbase'?"
>
> I like it!
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Jan 12, 2009 at 11:20 PM, Sunny Hirai <th...@gmail.com> wrote:
> Thanks for all the responses.
>
> Just to clarify, I know that CouchDB is not relational and I know the
> primary differences and limitations; however, I still have some questions if
> you will permit.
>
We permit questions. :) Especially well thought out ones like you have below.
> While it could be noted as a weakness of my implementation, the other thing
> to note is that _id can then no longer be used generically. For example, I
> can not include a reference from another SQL database unless I make the
> reference a very long String encoded with all the unique values which seems
> to be a bad way to relate tables/databases.
>
Hence the MD5 suggestion. It's a length limited string that (while
probablistically, similar to UUIDs) guarantee global identity.
> Note that there is no way to handle two unique fields (e.g. "name" and
> "email" both unique).
>
Well, MD5 the string representation of them would be fine.
(Assumptions are obvious)
> I know that CouchDB has different pros and cons from relational databases
> and I'm okay with there being cons. I just want to make sure that what I'm
> asking is (a) impossible to do because of the way CouchDB works or (b) a
> design choice that is not constrained by CouchDB.
>
> The reason I ask is that there appears to be some sort of a lock somewhere
> to assure that you don't end up with two documents with the same id. For
> example, if you PUT two new documents at exactly the same time in the same
> server, one of them will fail because it will not have a "_rev". This is
> actually an assertion, I'm not sure to be true.
>
Let me start with the fact that this is an assertion that is mostly
true and only became slightly untrue in the last week or so. More on
this below.
> I understand that two documents PUT into two different servers that
> replicate the same database can conflict; however, can two of the documents
> conflict in the same database?
>
> If they CAN conflict, then the guaranteed uniqueness on a single server is
> not actually guaranteed upon a successful insert. They can both "succeed"
> but they can be in conflict. This could be bad for doing things like
> creating a new user as two users can be granted the same name successfully
> but only one actually gets it. In this case, the solutions to use _id to
> guarantee a unique name can actually fail anyways, even though it may be
> rare.
>
You're hitting on a part of the implementation that probably solely
resides in Damien's head at this moment. I haven't seen the end
implementation so I'll only be able to give you my best guess on what
will happen in the coming days/weeks/months.
> On the other hand, if _id CANNOT conflict within the same server, then it
> appears there is some sort of lock somewhere. It might be very light, or
> small, or whatever, but then there is a lock.
>
Its optional now with the "X-Couch-Full-Commit: true" header will
ensure a full commit.
> So, in other words, I would like to know which one is true:
>
> A. there can be conflicts _id conflicts on the same server. In that case,
> _id doesn't guarantee uniqueness in the sense that two records can be
> inserted successfuly, but only one is authoritative. Then I have to deal
> with this somehow anyways.
>
> B. there aren't conflicts on the same server so you are guaranteed
> uniqueness on the same server. The _id hack always works. In this case could
> we not consider a similar situation to guarantee unique fields, perhaps in
> the far (far) future? Even if not, I'd like to know that there can be no
> conflicts on the same server.
>
> C. Something else completely that allows both a conflict-free _id in a
> manner that is simultaneously lock free that I haven't thought of.
>
This is all from memory without reading or using any of this new code
yet, but the situation is something like the following. Remember, I'm
not entirely certain on all these things, its 11:33, and I've had
beers. Please no pointing and laughing.
Briefly:
Old school style: Single node couchdb ensured global uniqueness when
using PUT. When using POST to _bulk_docs there were transactional
semantics, if one of the docs failed all failed.
New school style: Giving transactional semantics on _bulk_docs is
inefficient to do when contemplating multi-node setups. CouchDB
multi-node setups refers to having the transparent Couch automagically
hashes documents and distributes accordingly.
Uncertain style: Damien commited code to make the transaction
semantics optional using a header for the request. This was presented
in terms of _bulk_docs. I have no idea how it affects PUT semantics on
a single node or otherwise.
Certainly muddy waters uncertain style: Given that I have no idea on
the specifics of the header flag, if it's specified then I would be
running under the assumption that you will get a notification that
something at least conflicted or it might fail the request.
Moving on...
So the idea is that you're either going to wait for a possibly super
long time for a transaction, or write code that deals with conflicts.
The recommendation is that you write code that deals with conflicts.
I'm sure I futzed something in there, so wait for corrections before
you come to any grand conclusins ;)
HTH,
Paul Davis
> Thanks for the feedback.
>
> Sunny
>
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Sunny Hirai <th...@gmail.com>.
Thanks for all the responses.
Just to clarify, I know that CouchDB is not relational and I know the
primary differences and limitations; however, I still have some questions if
you will permit.
While it could be noted as a weakness of my implementation, the other thing
to note is that _id can then no longer be used generically. For example, I
can not include a reference from another SQL database unless I make the
reference a very long String encoded with all the unique values which seems
to be a bad way to relate tables/databases.
Note that there is no way to handle two unique fields (e.g. "name" and
"email" both unique).
I know that CouchDB has different pros and cons from relational databases
and I'm okay with there being cons. I just want to make sure that what I'm
asking is (a) impossible to do because of the way CouchDB works or (b) a
design choice that is not constrained by CouchDB.
The reason I ask is that there appears to be some sort of a lock somewhere
to assure that you don't end up with two documents with the same id. For
example, if you PUT two new documents at exactly the same time in the same
server, one of them will fail because it will not have a "_rev". This is
actually an assertion, I'm not sure to be true.
I understand that two documents PUT into two different servers that
replicate the same database can conflict; however, can two of the documents
conflict in the same database?
If they CAN conflict, then the guaranteed uniqueness on a single server is
not actually guaranteed upon a successful insert. They can both "succeed"
but they can be in conflict. This could be bad for doing things like
creating a new user as two users can be granted the same name successfully
but only one actually gets it. In this case, the solutions to use _id to
guarantee a unique name can actually fail anyways, even though it may be
rare.
On the other hand, if _id CANNOT conflict within the same server, then it
appears there is some sort of lock somewhere. It might be very light, or
small, or whatever, but then there is a lock.
So, in other words, I would like to know which one is true:
A. there can be conflicts _id conflicts on the same server. In that case,
_id doesn't guarantee uniqueness in the sense that two records can be
inserted successfuly, but only one is authoritative. Then I have to deal
with this somehow anyways.
B. there aren't conflicts on the same server so you are guaranteed
uniqueness on the same server. The _id hack always works. In this case could
we not consider a similar situation to guarantee unique fields, perhaps in
the far (far) future? Even if not, I'd like to know that there can be no
conflicts on the same server.
C. Something else completely that allows both a conflict-free _id in a
manner that is simultaneously lock free that I haven't thought of.
Thanks for the feedback.
Sunny
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Dean Landolt <de...@deanlandolt.com>.
On Mon, Jan 12, 2009 at 10:04 PM, Antony Blakey <an...@gmail.com>wrote:
>
> On 13/01/2009, at 1:25 PM, Chris Anderson wrote:
>
> We're taking back the word "database"!
>>
>
> Yeah, right - not gonna happen :) More than any other issue - _slow_view
> etc - this induces assumptions that lead to confusion, frustration, endless
> support/help problems.
>
> I vote for Couch Document Store!
>
The word document is just as overloaded as database. I say Couch Data Store
(datastore ~ database without the assumptions).
Or like everything else, document the hell out of it and make heavy use of
the RTFM acronym.
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Antony Blakey <an...@gmail.com>.
On 14/01/2009, at 4:52 AM, Chris Anderson wrote:
> On Tue, Jan 13, 2009 at 3:44 AM, Samuel Flores <sa...@gmail.com>
> wrote:
>> I really don't understand the discussion. Documents are data too. If
>> "database" refer only to relational data then there's no need for
>> the 'R'
>> in RDBMS.
>>
>
> Oh I don't think we'd seriously rename anything based on this line of
> thought, but it is amusing.
Unlike _slow_view?
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
All that is required for evil to triumph is that good men do nothing.
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Chris Anderson <jc...@gmail.com>.
On Tue, Jan 13, 2009 at 3:44 AM, Samuel Flores <sa...@gmail.com> wrote:
> I really don't understand the discussion. Documents are data too. If
> "database" refer only to relational data then there's no need for the 'R'
> in RDBMS.
>
Oh I don't think we'd seriously rename anything based on this line of
thought, but it is amusing.
--
Chris Anderson
http://jchris.mfdz.com
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Sven Helmberger <sv...@gmx.de>.
Samuel Flores schrieb:
> I really don't understand the discussion. Documents are data too. If
> "database" refer only to relational data then there's no need for the
> 'R' in RDBMS.
>
From a logical point of view I totally agree. Unfortunately RDBMS have
dominated for so long that they have become the default database
technology in a lot of people's mind.
Database == RDBMS
Regards,
Sven Helmberger
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Samuel Flores <sa...@gmail.com>.
I really don't understand the discussion. Documents are data too. If
"database" refer only to relational data then there's no need for the
'R' in RDBMS.
Samuel Flores
On 13/01/2009, at 01:21, "Chris Anderson" <jc...@gmail.com> wrote:
> On Mon, Jan 12, 2009 at 7:51 PM, Jason Huggins <ja...@jrandolph.com>
> wrote:
>
>> "Database? Don't you mean 'docbase'?"
>
> I like it!
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Chris Anderson <jc...@gmail.com>.
On Mon, Jan 12, 2009 at 7:51 PM, Jason Huggins <ja...@jrandolph.com> wrote:
> "Database? Don't you mean 'docbase'?"
I like it!
--
Chris Anderson
http://jchris.mfdz.com
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Jan 12, 2009 at 10:51 PM, Jason Huggins <ja...@jrandolph.com> wrote:
>> On 13/01/2009, at 1:25 PM, Chris Anderson wrote:
>>> We're taking back the word "database"!
>
> On Mon, Jan 12, 2009 at 9:04 PM, Antony Blakey wrote:
>> I vote for Couch Document Store!
>
> I think "Couch Document Base"
Cause all your base... enh, that's as far as I can bring myself....
would be a nice little backronym to
> start using in the twitterverse/blogosphere/internets. And let's use
> the phrase "Database? Don't you mean 'docbase'?" as a kind of secret
> codephrase between friends on IRC or at the pub or at
> EuroCouchDBExpoCon 2009. Or simply use it on the back of all official
> CouchDB t-shirts from now on. I don't know whether to insert a smiley
> right about now or whether I'm serious... but I'm leaning towards
> serious.
>
> cheers,
> hugs
>
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Jason Huggins <ja...@jrandolph.com>.
> On 13/01/2009, at 1:25 PM, Chris Anderson wrote:
>> We're taking back the word "database"!
On Mon, Jan 12, 2009 at 9:04 PM, Antony Blakey wrote:
> I vote for Couch Document Store!
I think "Couch Document Base" would be a nice little backronym to
start using in the twitterverse/blogosphere/internets. And let's use
the phrase "Database? Don't you mean 'docbase'?" as a kind of secret
codephrase between friends on IRC or at the pub or at
EuroCouchDBExpoCon 2009. Or simply use it on the back of all official
CouchDB t-shirts from now on. I don't know whether to insert a smiley
right about now or whether I'm serious... but I'm leaning towards
serious.
cheers,
hugs
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Antony Blakey <an...@gmail.com>.
On 13/01/2009, at 1:25 PM, Chris Anderson wrote:
> We're taking back the word "database"!
Yeah, right - not gonna happen :) More than any other issue -
_slow_view etc - this induces assumptions that lead to confusion,
frustration, endless support/help problems.
I vote for Couch Document Store!
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Plurality is not to be assumed without necessity
-- William of Ockham (ca. 1285-1349)
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Chris Anderson <jc...@gmail.com>.
On Mon, Jan 12, 2009 at 5:43 PM, Antony Blakey <an...@gmail.com> wrote:
> CouchDB is so far from a relation database that even including the word
> 'database' (DB), with all of it's presumptive extensions, is unfortunate
> IMO.
We're taking back the word "database"!
:)
--
Chris Anderson
http://jchris.mfdz.com
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Antony Blakey <an...@gmail.com>.
On 13/01/2009, at 11:42 AM, James Marca wrote:
> You are restricting the use of _id for this "feature" due to your
> design decision of using just integer ids. But I don't think you can
> also complain that there is no way to guarantee the uniqueness of an
> arbitrary (set of) field(s). There is, you're just not using it.
> Again, I haven't been using couchdb for more than two months, but I'd
> guess the developers are not going to see a need to allow for
> *another* guaranteed unique field.
Particularly given that the uniqueness of the id field is to do with
document identity, rather than uniqueness per-se.
A fundamental issue you have to remember is that Couch is designed to
act in a disconnected p2p fashion. Different peers can create and edit
documents, and when they replicate you get conflicts that have to be
handled without the benefit of any global conflict resolution. This is
why considerations of uniqueness are a dead-end. You have to think in
terms of document identity, and the ontological mapping of that
identity. If identity in your domain is determined by the name you
give something, then that should be the id, and if you use writable
replicas then you need to handle replication conflicts. So uniqueness
in the sense that you are looking for is just not part of Couch's big
picture. As soon as the concept of 'lock' appears in your thinking,
you know you're on the wrong track.
As others have said, if you want a global state you need to look for
something outside of Couch, and that has it's own problems because
global state in a distributed manner is dependent on connectivity.
CouchDB is so far from a relation database that even including the
word 'database' (DB), with all of it's presumptive extensions, is
unfortunate IMO.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Isn't it enough to see that a garden is beautiful without having to
believe that there are fairies at the bottom of it too?
-- Douglas Adams
Re: Can I guarantee uniqueness in a field without using _id?
Posted by James Marca <jm...@translab.its.uci.edu>.
On Mon, Jan 12, 2009 at 03:05:32PM -0800, Sunny Hirai wrote:
> >
> > You have just identified the problem and solution that CouchDB solves
> > with using a traditional RDBMS in a distributed environment. Check out
> > the discussion on the CAP hypothesis (Others call it a theorem, I'm
> > not convinced, but the basic idea is there).
> >
> > Or more succinctly, guaranteed uniqueness requires global state,
> > global state does not scale.
> >
> > HTH,
> > Paul Davis
> >
>
> Yeah, that's what I thought but I was hoping for something in between.
>
> That is, all I want to do is guarantee uniqueness on one or a set of fields.
> It seems to me that it would not require locks on reads and most of eventual
> consistency would be okay except for when dealing with an INSERT or an
> UPDATE that changes the unique field. All other cases could work just like
> before. It would only require a special locked lookup on the unique field or
> field set which could be optimized.
>
> I know that CouchDB is not a solution to all problems (nor do I think it
> should be) but it seems that this would be common for most web applications
> which is a major CouchDB target. For example, almost any website needs
> unique user_names. They also need unique names for pages in a wiki. Blogs
> that use named pages would also have this problem.
>
> In other words, this seems like an issue that happens so often in the web
> computing world that I wonder if it should be supported in CouchDB.
>
But ... you pointed out that CouchDB can already provide uniqueness,
as long as you use _id for this. In my application wherever I need to
guarantee a unique document field (in my case, data from a particular
site at a particular time) I just concatenate those fields with some
known field separator to make that the _id of the document.
You are restricting the use of _id for this "feature" due to your
design decision of using just integer ids. But I don't think you can
also complain that there is no way to guarantee the uniqueness of an
arbitrary (set of) field(s). There is, you're just not using it.
Again, I haven't been using couchdb for more than two months, but I'd
guess the developers are not going to see a need to allow for
*another* guaranteed unique field.
But before you think this post is just a flame or something, I *agree*
with you that some way of asking for uniqueness is a perfectly good
feature to have. On the data upload side, I can see flagging unique
data using something trivial like '_unique/foo':{'key1':value,
'key2':[...],...} to indicate unique data to the engine in the input
JSON (following the design document convention). It always strikes me
as a bit of a hack to overload the _id just to use its uniqueness
features. But given that one *can* get unique fields, I'd guess there
are other things for the developers to solve first.
Back to the problem at hand, perhaps you could translate whatever you
need to be guaranteed unique into integers in a one to one, reversible
mapping function. The function would only have to be more efficient
than writing wrapper code to handle non-integer _id values. Use
character codes or something.
Just my two cents,
James
> Of course, the argument is to push that global state into the application
> layer and store it elsewhere; however, the question is whether that
> complexity would better service the developer if it was (a) in CouchDB or
> (b) in the application itself.
>
> Since it is a common case (as opposed to an edge case), perhaps there could
> be some discussion of putting it into CouchDB.
>
> Seriously though, would this not affect 100% of websites with a user base?
> One solution, I suppose, is to store the user data in an SQL database, but I
> prefer having as much as possible in CouchDB due to its flexible schemas.
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
RE: Can I guarantee uniqueness in a field without using _id?
Posted by Kurt Mackey <ku...@arstechnica.com>.
I don't know if there's ever been any discussion of redirects for a given document ID, but that's one potential way Couch could handle something like this.
It might go something like this:
* Create a user, user gets a UUID as an identifier
* Create a redirect to user using key user-name-myusername
* Profit
You'd have to be clever about how you're creating users, maybe by generating a UUID first, then saving the unique redirect keys, then finally creating the user document.
-Kurt
-----Original Message-----
From: Sunny Hirai [mailto:thesunny@gmail.com]
Sent: Monday, January 12, 2009 5:06 PM
To: user@couchdb.apache.org
Subject: Re: Can I guarantee uniqueness in a field without using _id?
>
> You have just identified the problem and solution that CouchDB solves
> with using a traditional RDBMS in a distributed environment. Check out
> the discussion on the CAP hypothesis (Others call it a theorem, I'm
> not convinced, but the basic idea is there).
>
> Or more succinctly, guaranteed uniqueness requires global state,
> global state does not scale.
>
> HTH,
> Paul Davis
>
Yeah, that's what I thought but I was hoping for something in between.
That is, all I want to do is guarantee uniqueness on one or a set of fields.
It seems to me that it would not require locks on reads and most of eventual
consistency would be okay except for when dealing with an INSERT or an
UPDATE that changes the unique field. All other cases could work just like
before. It would only require a special locked lookup on the unique field or
field set which could be optimized.
I know that CouchDB is not a solution to all problems (nor do I think it
should be) but it seems that this would be common for most web applications
which is a major CouchDB target. For example, almost any website needs
unique user_names. They also need unique names for pages in a wiki. Blogs
that use named pages would also have this problem.
In other words, this seems like an issue that happens so often in the web
computing world that I wonder if it should be supported in CouchDB.
Of course, the argument is to push that global state into the application
layer and store it elsewhere; however, the question is whether that
complexity would better service the developer if it was (a) in CouchDB or
(b) in the application itself.
Since it is a common case (as opposed to an edge case), perhaps there could
be some discussion of putting it into CouchDB.
Seriously though, would this not affect 100% of websites with a user base?
One solution, I suppose, is to store the user data in an SQL database, but I
prefer having as much as possible in CouchDB due to its flexible schemas.
Sunny
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Sunny Hirai <th...@gmail.com>.
>
> You have just identified the problem and solution that CouchDB solves
> with using a traditional RDBMS in a distributed environment. Check out
> the discussion on the CAP hypothesis (Others call it a theorem, I'm
> not convinced, but the basic idea is there).
>
> Or more succinctly, guaranteed uniqueness requires global state,
> global state does not scale.
>
> HTH,
> Paul Davis
>
Yeah, that's what I thought but I was hoping for something in between.
That is, all I want to do is guarantee uniqueness on one or a set of fields.
It seems to me that it would not require locks on reads and most of eventual
consistency would be okay except for when dealing with an INSERT or an
UPDATE that changes the unique field. All other cases could work just like
before. It would only require a special locked lookup on the unique field or
field set which could be optimized.
I know that CouchDB is not a solution to all problems (nor do I think it
should be) but it seems that this would be common for most web applications
which is a major CouchDB target. For example, almost any website needs
unique user_names. They also need unique names for pages in a wiki. Blogs
that use named pages would also have this problem.
In other words, this seems like an issue that happens so often in the web
computing world that I wonder if it should be supported in CouchDB.
Of course, the argument is to push that global state into the application
layer and store it elsewhere; however, the question is whether that
complexity would better service the developer if it was (a) in CouchDB or
(b) in the application itself.
Since it is a common case (as opposed to an edge case), perhaps there could
be some discussion of putting it into CouchDB.
Seriously though, would this not affect 100% of websites with a user base?
One solution, I suppose, is to store the user data in an SQL database, but I
prefer having as much as possible in CouchDB due to its flexible schemas.
Sunny
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Paul Davis <pa...@gmail.com>.
> Unless anybody can think of a way to guarantee uniqueness in CouchDB, I
> think I am going to abandon the notion of GUARANTEED uniqueness and instead
> implementcode to PREFER uniqueness. Then at the end of the day, we can write
> some code that will tell us if that 1 in a million chance (i.e. two
> simultaneous inserts/updates) left our database in a state where there are
> two docs with the same unique field.
>
> Since the odds are quite low that it will actually happen, I think we can
> deal with the times that it happens manually so long as we know about it.
>
> Sunny
>
You have just identified the problem and solution that CouchDB solves
with using a traditional RDBMS in a distributed environment. Check out
the discussion on the CAP hypothesis (Others call it a theorem, I'm
not convinced, but the basic idea is there).
Or more succinctly, guaranteed uniqueness requires global state,
global state does not scale.
HTH,
Paul Davis
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Sunny Hirai <th...@gmail.com>.
>
>
> I know I can put the 'name' in the '_id' field but I'd like to avoid it
>> because I'm abstracting CouchDB and the abstraction does not allow for it.
>>
>
> Why is that?
>
Amongst other things, the framework can join CouchDB tables to SQL tables.
To allow this to happen, we need to be able to easily share references
through an id field. I use a :bigint like field to handle this. Not
consistently knowing that the id is a :bigint will incur a lot of code
overhead where the code is very clean otherwise. In some cases, we also need
to have uniqueness based on two fields. This would be even harder to store
as the lone _id field, particularly since CouchDB seems to convert all ids
to String.
But, because of the ability to join between two database types, we can use
the database that is the best fit for our needs with more or less the same
code.
For example:
* Where schema doesn't change often and it has to do with transactional
data, we prefer SQL. A good example might be data to do with money
transactions.
* Where schema needs to be updated often and the data is not very
transactional, we prefer CouchDB. For example, pages in a website. Over
time, we might add tags, or descriptions to those pages. We also might
introduce new page types, some of which support different fields. In this
case, CouchDB is preferred.
Note that models are defined and the API is almost exactly the same between
CouchDB and SQL. The main difference is we define and use "indexes" in SQL
and "views" in CouchDB. It uses similar syntax to Ruby Sequel (and in fact
uses Sequel as a backend for the SQL API).
>
> Anybody have any solutions other than making 'name' the '_id' field?
>>
>
> You'd need some service that acts as global state outside of CouchDB
> that ensures it only allows one client to take a certain name. CouchDB
> won't do that for you.
>
> Cheers
> Jan
> --
>
I wanted to do this automatically somehow. I considered storing unique
values in an outside SQL table just for those unique fields and have it
check automatically against that. Unfortunately, there still seems to be
edge cases that would make this inconsistent. For example, if during the
commit, one or more databases failed. Of course, we could work around those
as well, but now there is a lot of stuff going on to do an insert or update
that messies up the code. I don't like the junk it adds.
Unless anybody can think of a way to guarantee uniqueness in CouchDB, I
think I am going to abandon the notion of GUARANTEED uniqueness and instead
implementcode to PREFER uniqueness. Then at the end of the day, we can write
some code that will tell us if that 1 in a million chance (i.e. two
simultaneous inserts/updates) left our database in a state where there are
two docs with the same unique field.
Since the odds are quite low that it will actually happen, I think we can
deal with the times that it happens manually so long as we know about it.
Sunny
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Jan Lehnardt <ja...@apache.org>.
On 12 Jan 2009, at 00:30, Sunny Hirai wrote:
> I know I can put the 'name' in the '_id' field but I'd like to avoid
> it
> because I'm abstracting CouchDB and the abstraction does not allow
> for it.
Why is that?
> Anybody have any solutions other than making 'name' the '_id' field?
You'd need some service that acts as global state outside of CouchDB
that ensures it only allows one client to take a certain name. CouchDB
won't do that for you.
Cheers
Jan
--
Re: Can I guarantee uniqueness in a field without using _id?
Posted by "ara.t.howard" <ar...@gmail.com>.
On Jan 13, 2009, at 11:51 PM, Chris Anderson wrote:
> CouchDB will enforce unique docids within a database on a single node.
> In the long run, we may also support partitioning a single database
> across many nodes. In that case, we should also support uniqueness
> constraints with a single logical database. So should your name server
> get too busy for one box, Couch should support spreading the load
> across a cluster of boxes.
even more useful would be to be able to set some sort of key prefix
per server. a scope if you will.
that plus ha-proxy to multiple backends would be powerful.
a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Chris Anderson <jc...@gmail.com>.
On Tue, Jan 13, 2009 at 6:42 PM, Sunny Hirai <th...@gmail.com> wrote:
> Ara.T.Howard and Antony Blakey:
>
> Thanks for the feedback.
>
> Hmm, this idea is interesting assuming that CouchDB will enforce the
> uniqueness per database (see earlier post I made).
CouchDB will enforce unique docids within a database on a single node.
In the long run, we may also support partitioning a single database
across many nodes. In that case, we should also support uniqueness
constraints with a single logical database. So should your name server
get too busy for one box, Couch should support spreading the load
across a cluster of boxes.
>
> I think I will consider this.
>
> In-ter-esting.
>
> Sunny
>
--
Chris Anderson
http://jchris.mfdz.com
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Sunny Hirai <th...@gmail.com>.
Ara.T.Howard and Antony Blakey:
Thanks for the feedback.
Hmm, this idea is interesting assuming that CouchDB will enforce the
uniqueness per database (see earlier post I made).
I think I will consider this.
In-ter-esting.
Sunny
Re: Can I guarantee uniqueness in a field without using _id?
Posted by Antony Blakey <an...@gmail.com>.
On 14/01/2009, at 11:40 AM, ara.t.howard wrote:
> assume your system will be distributed at some point and that global
> uniqueness is hard whether in a rdbms or docms. even auto-increment
> will fail here. now take the middle path - only writes need to be
> unique because, if all of them are, objects can distribute between
> nodes freely (since we already know they are unique inside the
> system).
>
> so just make a rule - you will require a single couchdb instance to
> be available to check uniqueness, but any couchdb instance up to
> perform reads or writes which do not need to be unique (like
> comments with a timestamp or something).
>
> so host a couchdb at something like
>
> http://domain.org/unique
>
> use this to check the uniqueness of fields. for instance, so see if
> a name is in the system try to put
>
> { '_id' : 'name::yourname' }
>
> or something you app will use to decide the uniqueness of a name in
> the system. think of it as a global hash you can check for a given
> key.
>
> to me this seems quite reasonable, it's a nod to the fact that
> distributed uniqueness is very hard, while at the same time
> accepting that most applications are difficult to build without it.
> the constraint on your system will simply be that the uniqueness
> server must be up for some operations to succeed, but if you design
> for it many will be able to perform many other operations in the
> face of a uniqueness server failure.
I am doing a similar thing - many read-replicas and one write replica.
My domain tolerates unavailability of write-access, but not read-
access. The application is distributed with each couch instance, and
as soon as a user logs on with write access, they are switched to a
unique write server.
I switch the user on logon to the write server, for all access,
because otherwise they don't get any locally consistent view. Hence I
don't distinguish between data-needing-uniqueness or not.
For each domain object, I determine the name used to identify it
within it's parent context e.g. a user is an email address, a page in
a CMS is a parent id + name. I then use that identity, either with a
prefix (e.g. user:antony.blakey@...) or in a separate db, as the
document id. As soon as I use a prefix on any data in a given db, I
use a prefix on all data in that db that uses domain keys on ids.
As an aside, for hierarchic data (which covers any heterogenous data
with a qualified name/uniqueness constraint), if you have a single
write point then you can assign a UUID to each node when it is
created, and presuming that field is immutable, you can then qualify
the child node using the parent's UUID rather than the parent's id
(which can grow in a nasty fashion).
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The greatest challenge to any thinker is stating the problem in a way
that will allow a solution
-- Bertrand Russell
Re: Can I guarantee uniqueness in a field without using _id?
Posted by "ara.t.howard" <ar...@gmail.com>.
On Jan 11, 2009, at 4:30 PM, Sunny Hirai wrote:
> First of all, I'd like to say that CouchDB is an amazing project.
> I've been
> following it for a year and have recently made it a first class
> citizen of
> my framework.
>
> Anyways, looked for this answer but I couldn't find it.
>
> Let's say I want to create a database that contains the pages in a
> wiki.
>
> Each document looks something like this:
>
> {
> name: 'about_us',
> body: 'This is the about us page'
> }
>
> There would be a view on the database where the key is 'name'.
>
> The problem is this:
>
> User A goes to create the 'aboutus' page. We want it to be unique so
> we
> check if it is in the view. Since it is the first 'about_us' page,
> we can't
> find it. So we are about to do the insert but haven't started yet.
>
> In the meantime, User B goes to create the 'aboutus' page. We check
> the view
> and it also isn't there yet because User A has not completed its
> insert. We
> start to do the insert for User B.
>
> In this case, both User A and User B have begun the insert process.
>
> At the end of the inserts, we will have two documents with the same
> name
> 'aboutus'. Can this be prevented?
>
> I know I can put the 'name' in the '_id' field but I'd like to avoid
> it
> because I'm abstracting CouchDB and the abstraction does not allow
> for it.
> Anybody have any solutions other than making 'name' the '_id' field?
>
> -Sunny
this is what i'm doing:
assume your system will be distributed at some point and that global
uniqueness is hard whether in a rdbms or docms. even auto-increment
will fail here. now take the middle path - only writes need to be
unique because, if all of them are, objects can distribute between
nodes freely (since we already know they are unique inside the system).
so just make a rule - you will require a single couchdb instance to be
available to check uniqueness, but any couchdb instance up to perform
reads or writes which do not need to be unique (like comments with a
timestamp or something).
so host a couchdb at something like
http://domain.org/unique
use this to check the uniqueness of fields. for instance, so see if a
name is in the system try to put
{ '_id' : 'name::yourname' }
or something you app will use to decide the uniqueness of a name in
the system. think of it as a global hash you can check for a given key.
to me this seems quite reasonable, it's a nod to the fact that
distributed uniqueness is very hard, while at the same time accepting
that most applications are difficult to build without it. the
constraint on your system will simply be that the uniqueness server
must be up for some operations to succeed, but if you design for it
many will be able to perform many other operations in the face of a
uniqueness server failure.
my 2cts.
a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama