You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Luis Miguel Silva <lu...@gmail.com> on 2011/04/05 23:46:10 UTC
Update conflicts?
Dear all,
I'm trying to play around with updates and i'm bumping into some problems.
Let's image we have to clients that poll a document from the server at
the same time and get the same _rev.
Then one of them updates the doc based on the _rev it got:
[root@xkitten ~]# curl -X PUT -d
'{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
http://localhost:5984/benchmark/test?conflicts=true
{"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
Then another one tries to update the doc based on the same exact _rev:
[root@xkitten ~]# curl -X PUT -d
'{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
http://localhost:5984/benchmark/test?conflicts=true
{"error":"conflict","reason":"Document update conflict."}
[root@xkitten ~]#
Is there a way to avoid this?! (like...make the update just create a
new _rev or something)??
Ideally, we would be able to update without specifying the _rev, just
posting (or, in this case PUTting) to the document...
Thoughts??
Thank you,
Luis
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
heheh, well, i'm creating a cloud solution for others to use when they
can't architect their own stuff so...i think i have an excuse! ;o)
Essentially, my problem is not just solving these "small problems"
but, instead, maintaining them in an elegant way others can use / take
advantage off.
CouchDB would be a part of our cloud solution and we would need people
to be able to very easily adapt the database / change it according to
their own needs - and THAT is very hard to productize...
Plus, there is definitely a learning curve in CouchDB! I'm very
excited about the features it has but, every time i write to this
list, someone points me to a brand new thing i had never heard about
before :oP
- bulk_docs api
- changes api
- views
- lists
- updater functions
- ...
It's exciting to find out about these things BUT overwhelming ;o)
On Wed, Apr 6, 2011 at 10:10 AM, kowsik <ko...@gmail.com> wrote:
> On Wed, Apr 6, 2011 at 8:37 AM, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>> Thank you all very much for all your suggestions!
>> I'm going to report to my company that, in theory, we can overcome the
>> problems we are encountering and i'll see what they want to do (either
>> continue investing in CouchDB research, research MongoDB or develop
>> our own solution from scratch)...
>
> Sorry, couldn't resist:
> http://twitter.com/DEVOPS_BORAT/status/55074539922653184 :)
>
> K.
> ---
> http://blitz.io
> http://twitter.com/pcapr
>
Re: Update conflicts?
Posted by kowsik <ko...@gmail.com>.
On Wed, Apr 6, 2011 at 8:37 AM, Luis Miguel Silva
<lu...@gmail.com> wrote:
> Thank you all very much for all your suggestions!
> I'm going to report to my company that, in theory, we can overcome the
> problems we are encountering and i'll see what they want to do (either
> continue investing in CouchDB research, research MongoDB or develop
> our own solution from scratch)...
Sorry, couldn't resist:
http://twitter.com/DEVOPS_BORAT/status/55074539922653184 :)
K.
---
http://blitz.io
http://twitter.com/pcapr
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
Thank you all very much for all your suggestions!
I'm going to report to my company that, in theory, we can overcome the
problems we are encountering and i'll see what they want to do (either
continue investing in CouchDB research, research MongoDB or develop
our own solution from scratch)...
Again, thank you very much for your time! :o)
Luis
On Wed, Apr 6, 2011 at 9:11 AM, Sean Copenhaver
<se...@gmail.com> wrote:
> I thought list and show functions could return whatever format of data you
> wanted? Not just HTML.
>
> So you could split out the document into pieces to avoid the conflicts and
> use a view to retrieve the related bits, then a list function to generate
> out the final json document. At least that's my understanding.
>
> http://guide.couchdb.org/draft/transforming.html
>
> Also I would be careful about the forcing conflicts. The document you want
> may not be picked as the winner and if you are quickly relying on it that
> could cause some undesirable effects. I would suggest understanding how
> conflict "winners" are picked by couch. I personally only have a high level
> understanding of this whole conflict resolution and the effects of such, but
> the wiki has some information. You may want to check out the "View Map
> Functions" and the following sections:
>
> http://wiki.apache.org/couchdb/Replication_and_conflicts
>
> I honestly think you might want to tackle the problem with document design
> and transformation with show/list functions first though.
>
> On Wed, Apr 6, 2011 at 8:20 AM, Matt Goodall <ma...@gmail.com> wrote:
>
>> On 5 April 2011 22:46, Luis Miguel Silva
>> <lu...@gmail.com>wrote:
>>
>> > Dear all,
>> >
>> > I'm trying to play around with updates and i'm bumping into some
>> problems.
>> >
>> > Let's image we have to clients that poll a document from the server at
>> > the same time and get the same _rev.
>> > Then one of them updates the doc based on the _rev it got:
>> > [root@xkitten ~]# curl -X PUT -d
>> > '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>> > http://localhost:5984/benchmark/test?conflicts=true
>> > {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>> > Then another one tries to update the doc based on the same exact _rev:
>> > [root@xkitten ~]# curl -X PUT -d
>> > '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>> > http://localhost:5984/benchmark/test?conflicts=true
>> > {"error":"conflict","reason":"Document update conflict."}
>> > [root@xkitten ~]#
>> >
>> > Is there a way to avoid this?! (like...make the update just create a
>> > new _rev or something)??
>> >
>> > Ideally, we would be able to update without specifying the _rev, just
>> > posting (or, in this case PUTting) to the document...
>> >
>> > Thoughts??
>> >
>> >
>> I hope I haven't missed this suggestion in the thread, but here goes ...
>>
>> Another option, depending on your application, is to "force" the update to
>> the document through and worry about conflicts later.
>>
>> If you use CouchDB's bulk update facility with the all_or_nothing=true
>> option then your document will always be stored but may be marked as a
>> conflict in the database. When a conflict does happen CouchDB chooses a
>> winning version and stores the rev(s) of the loser(s) in the document's
>> _conflicts attribute. But whatever happens, the update will not return an
>> error to your application.
>>
>> Depending on the application conflicts can then be ignored or require
>> intervention to fix. Sometimes the fix can be automated. Conflicts are easy
>> to find: either poll a view that emits any doc with a _conflicts attribute
>> or watch a _changes feed with a filter that returns only the docs with
>> _conflicts.
>>
>> Finally, once you know the id of a document containing conflicts you can
>> find out exactly when the conflicts happened by GET'ing it with a
>> conflicts=true option and inspecting the _conflicts attribute.
>>
>> I realise there's a lot to take in there so the following wiki pages are
>> probably essential reading:
>>
>> * Bulk update API, http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API.
>> Specifically the "Transactional Semantics with Bulk Updates" section.
>>
>> * Document API, http://wiki.apache.org/couchdb/HTTP_Document_API. Contains
>> information on the conflicts=true option and the attributes in a document
>> that describe the conflicting versions.
>>
>> * Changes API, http://wiki.apache.org/couchdb/HTTP_database_API#Changes.
>>
>>
>> Note: One advantage of using CouchDB like this is that once you start using
>> replication these conflicts can happen anyway. However, you've already
>> solved the problem of resolving conflicts so it doesn't really make any
>> difference because your application was written to expect conflicts from
>> the
>> start.
>>
>>
>> Hope that all makes some sense!
>>
>>
>> - Matt
>>
>
>
>
> --
> “The limits of language are the limits of one's world. “ -Ludwig von
> Wittgenstein
>
Re: Update conflicts?
Posted by Sean Copenhaver <se...@gmail.com>.
I thought list and show functions could return whatever format of data you
wanted? Not just HTML.
So you could split out the document into pieces to avoid the conflicts and
use a view to retrieve the related bits, then a list function to generate
out the final json document. At least that's my understanding.
http://guide.couchdb.org/draft/transforming.html
Also I would be careful about the forcing conflicts. The document you want
may not be picked as the winner and if you are quickly relying on it that
could cause some undesirable effects. I would suggest understanding how
conflict "winners" are picked by couch. I personally only have a high level
understanding of this whole conflict resolution and the effects of such, but
the wiki has some information. You may want to check out the "View Map
Functions" and the following sections:
http://wiki.apache.org/couchdb/Replication_and_conflicts
I honestly think you might want to tackle the problem with document design
and transformation with show/list functions first though.
On Wed, Apr 6, 2011 at 8:20 AM, Matt Goodall <ma...@gmail.com> wrote:
> On 5 April 2011 22:46, Luis Miguel Silva
> <lu...@gmail.com>wrote:
>
> > Dear all,
> >
> > I'm trying to play around with updates and i'm bumping into some
> problems.
> >
> > Let's image we have to clients that poll a document from the server at
> > the same time and get the same _rev.
> > Then one of them updates the doc based on the _rev it got:
> > [root@xkitten ~]# curl -X PUT -d
> > '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> > http://localhost:5984/benchmark/test?conflicts=true
> > {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> > Then another one tries to update the doc based on the same exact _rev:
> > [root@xkitten ~]# curl -X PUT -d
> > '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> > http://localhost:5984/benchmark/test?conflicts=true
> > {"error":"conflict","reason":"Document update conflict."}
> > [root@xkitten ~]#
> >
> > Is there a way to avoid this?! (like...make the update just create a
> > new _rev or something)??
> >
> > Ideally, we would be able to update without specifying the _rev, just
> > posting (or, in this case PUTting) to the document...
> >
> > Thoughts??
> >
> >
> I hope I haven't missed this suggestion in the thread, but here goes ...
>
> Another option, depending on your application, is to "force" the update to
> the document through and worry about conflicts later.
>
> If you use CouchDB's bulk update facility with the all_or_nothing=true
> option then your document will always be stored but may be marked as a
> conflict in the database. When a conflict does happen CouchDB chooses a
> winning version and stores the rev(s) of the loser(s) in the document's
> _conflicts attribute. But whatever happens, the update will not return an
> error to your application.
>
> Depending on the application conflicts can then be ignored or require
> intervention to fix. Sometimes the fix can be automated. Conflicts are easy
> to find: either poll a view that emits any doc with a _conflicts attribute
> or watch a _changes feed with a filter that returns only the docs with
> _conflicts.
>
> Finally, once you know the id of a document containing conflicts you can
> find out exactly when the conflicts happened by GET'ing it with a
> conflicts=true option and inspecting the _conflicts attribute.
>
> I realise there's a lot to take in there so the following wiki pages are
> probably essential reading:
>
> * Bulk update API, http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API.
> Specifically the "Transactional Semantics with Bulk Updates" section.
>
> * Document API, http://wiki.apache.org/couchdb/HTTP_Document_API. Contains
> information on the conflicts=true option and the attributes in a document
> that describe the conflicting versions.
>
> * Changes API, http://wiki.apache.org/couchdb/HTTP_database_API#Changes.
>
>
> Note: One advantage of using CouchDB like this is that once you start using
> replication these conflicts can happen anyway. However, you've already
> solved the problem of resolving conflicts so it doesn't really make any
> difference because your application was written to expect conflicts from
> the
> start.
>
>
> Hope that all makes some sense!
>
>
> - Matt
>
--
“The limits of language are the limits of one's world. “ -Ludwig von
Wittgenstein
Re: Update conflicts?
Posted by Matt Goodall <ma...@gmail.com>.
On 5 April 2011 22:46, Luis Miguel Silva
<lu...@gmail.com>wrote:
> Dear all,
>
> I'm trying to play around with updates and i'm bumping into some problems.
>
> Let's image we have to clients that poll a document from the server at
> the same time and get the same _rev.
> Then one of them updates the doc based on the _rev it got:
> [root@xkitten ~]# curl -X PUT -d
> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> http://localhost:5984/benchmark/test?conflicts=true
> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> Then another one tries to update the doc based on the same exact _rev:
> [root@xkitten ~]# curl -X PUT -d
> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> http://localhost:5984/benchmark/test?conflicts=true
> {"error":"conflict","reason":"Document update conflict."}
> [root@xkitten ~]#
>
> Is there a way to avoid this?! (like...make the update just create a
> new _rev or something)??
>
> Ideally, we would be able to update without specifying the _rev, just
> posting (or, in this case PUTting) to the document...
>
> Thoughts??
>
>
I hope I haven't missed this suggestion in the thread, but here goes ...
Another option, depending on your application, is to "force" the update to
the document through and worry about conflicts later.
If you use CouchDB's bulk update facility with the all_or_nothing=true
option then your document will always be stored but may be marked as a
conflict in the database. When a conflict does happen CouchDB chooses a
winning version and stores the rev(s) of the loser(s) in the document's
_conflicts attribute. But whatever happens, the update will not return an
error to your application.
Depending on the application conflicts can then be ignored or require
intervention to fix. Sometimes the fix can be automated. Conflicts are easy
to find: either poll a view that emits any doc with a _conflicts attribute
or watch a _changes feed with a filter that returns only the docs with
_conflicts.
Finally, once you know the id of a document containing conflicts you can
find out exactly when the conflicts happened by GET'ing it with a
conflicts=true option and inspecting the _conflicts attribute.
I realise there's a lot to take in there so the following wiki pages are
probably essential reading:
* Bulk update API, http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API.
Specifically the "Transactional Semantics with Bulk Updates" section.
* Document API, http://wiki.apache.org/couchdb/HTTP_Document_API. Contains
information on the conflicts=true option and the attributes in a document
that describe the conflicting versions.
* Changes API, http://wiki.apache.org/couchdb/HTTP_database_API#Changes.
Note: One advantage of using CouchDB like this is that once you start using
replication these conflicts can happen anyway. However, you've already
solved the problem of resolving conflicts so it doesn't really make any
difference because your application was written to expect conflicts from the
start.
Hope that all makes some sense!
- Matt
Re: Update conflicts?
Posted by Dennis Clark <db...@gmail.com>.
I believe (speaking somewhat out of turn here; I don't really know with
certainty) that update handlers can run in parallel. My understanding is
that they save documents in exactly the same way clients do, that is,
wholesale with a _rev -- it just happens that the update function gets the
old document complete with _rev when it's called. As such, if hypothetically
you had two updates, super_quick and takes_forever, and hit the second one
before the first one, the first one might finish first, leaving
takes_forever with a stale _rev by the time it tries to save.
I can't easily think of a use case where this actually matters, though,
given that update handlers tend to be computationally light.
On Tue, Apr 5, 2011 at 7:47 PM, Sean Copenhaver
<se...@gmail.com>wrote:
> Very cool. I honestly haven't had the need for updates yet so I only know
> that they exist. Good to know that you could basically do your own merging
> in an update.
>
> You mentioned that the first update function could finish during the
> second? I thought the writes in CouchDB were serial, so do update functions
> actually run in parallel and then the new/updated doc is put in the write
> queue?
>
> --
> Sean Copenhaver
>
> On Tuesday, April 5, 2011 at 7:32 PM, Dennis Clark wrote:
> > Actually no -- if you put to an update handler with an existing docid,
> the
> > handler gets a full copy of the then-current doc, _rev and all. As such,
> the
> > version conflict doesn't happen before the update function unless the
> first
> > update function completes during the operation of the second. This means
> > that you can use them for a counter (say) without requiring your clients
> to
> > know document _rev's, as in:
> >
> >
> http://stackoverflow.com/questions/2972068/couchdb-document-update-handlers-in-place-updates
> >
> > On Tue, Apr 5, 2011 at 7:19 PM, Sean Copenhaver
> > <se...@gmail.com>wrote:
> >
> > > Keep in mind that besides both being doc db's they are very different
> > > products.
> > >
> > > Also I'm not positive how far update functions can go. I believe the
> > > version conflict happens before your update function.
> > >
> > > You should evaluate if you are trying to use couchdb too much like the
> > > average relational db. Also think about if you can break up the docs in
> an
> > > appropriate way to avoid the conflicts.
> > >
> > >
> > >
> > > On Apr 5, 2011, at 7:12 PM, Dennis Clark <db...@gmail.com> wrote:
> > >
> > > > What's your actual function? Update functions should work fine for
> your
> > > use case, but can be a little finicky to get working, particularly the
> first
> > > time.
> > > >
> > > > On Apr 5, 2011, at 7:01 PM, Luis Miguel Silva <
> > > luismiguelferreirasilva@gmail.com> wrote:
> > > >
> > > > > Thank you! I'll probably have to evaluate it then...
> > > > >
> > > > > I've looked at the update handlers but i can't get them to work
> :o\...
> > > > > I'm creating a _design document with a "updates" field with an
> update
> > > > > function inside but i can't seem to get it to work.
> > > > >
> > > > > Either way, i think it's time i evaluate MongoDB :o((... (this was
> > > > > unexpected, i was completely sold on CouchDB :o|).
> > > > >
> > > > > On Tue, Apr 5, 2011 at 4:50 PM, Sean Copenhaver
> > > > > <se...@gmail.com> wrote:
> > > > > > Ah, the problem is that couchdb does not do partial updates. It
> writes
> > > the whole doc. MongoDB I believe does support partial updates though.
> No
> > > experience with it.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <
> > > luismiguelferreirasilva@gmail.com> wrote:
> > > > > >
> > > > > > > More or less!
> > > > > > >
> > > > > > > The most common scenario will be:
> > > > > > > - two or more processes writing to the same document, but only
> to a
> > > > > > > specific attribute (not overwriting the whole document)
> > > > > > >
> > > > > > > If, by any chance, two processes overwrite the same field, i'm
> ok with
> > > > > > > the last one always winning.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Luis
> > > > > > >
> > > > > > > On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <
> > > robert.newson@gmail.com> wrote:
> > > > > > > > "Ideally, we would be able to update without specifying the
> _rev,
> > > just
> > > > > > > > posting (or, in this case PUTting) to the document..."
> > > > > > > >
> > > > > > > > So you want to blindly overwrite some unknown data?
> > > > > > > >
> > > > > > > > B.
> > > > > > > >
> > > > > > > > On 5 April 2011 22:57, Zachary Zolton <
> zachary.zolton@gmail.com>
> > > wrote:
> > > > > > > > > Luis,
> > > > > > > > >
> > > > > > > > > Checkout _update handlers:
> > > > > > > > >
> > > > > > > > > http://wiki.apache.org/couchdb/Document_Update_Handlers
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Zach
> > > > > > > > >
> > > > > > > > > On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
> > > > > > > > > <lu...@gmail.com> wrote:
> > > > > > > > > > Dear all,
> > > > > > > > > >
> > > > > > > > > > I'm trying to play around with updates and i'm bumping
> into some
> > > problems.
> > > > > > > > > >
> > > > > > > > > > Let's image we have to clients that poll a document from
> the server
> > > at
> > > > > > > > > > the same time and get the same _rev.
> > > > > > > > > > Then one of them updates the doc based on the _rev it
> got:
> > > > > > > > > > [root@xkitten ~]# curl -X PUT -d
> > > > > > > > > >
> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> > > > > > > > > > http://localhost:5984/benchmark/test?conflicts=true
> > > > > > > > > >
> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> > > > > > > > > > Then another one tries to update the doc based on the
> same exact
> > > _rev:
> > > > > > > > > > [root@xkitten ~]# curl -X PUT -d
> > > > > > > > > >
> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> > > > > > > > > > http://localhost:5984/benchmark/test?conflicts=true
> > > > > > > > > > {"error":"conflict","reason":"Document update conflict."}
> > > > > > > > > > [root@xkitten ~]#
> > > > > > > > > >
> > > > > > > > > > Is there a way to avoid this?! (like...make the update
> just create
> > > a
> > > > > > > > > > new _rev or something)??
> > > > > > > > > >
> > > > > > > > > > Ideally, we would be able to update without specifying
> the _rev,
> > > just
> > > > > > > > > > posting (or, in this case PUTting) to the document...
> > > > > > > > > >
> > > > > > > > > > Thoughts??
> > > > > > > > > >
> > > > > > > > > > Thank you,
> > > > > > > > > > Luis
> >
>
Re: Update conflicts?
Posted by Sean Copenhaver <se...@gmail.com>.
Very cool. I honestly haven't had the need for updates yet so I only know that they exist. Good to know that you could basically do your own merging in an update.
You mentioned that the first update function could finish during the second? I thought the writes in CouchDB were serial, so do update functions actually run in parallel and then the new/updated doc is put in the write queue?
--
Sean Copenhaver
On Tuesday, April 5, 2011 at 7:32 PM, Dennis Clark wrote:
> Actually no -- if you put to an update handler with an existing docid, the
> handler gets a full copy of the then-current doc, _rev and all. As such, the
> version conflict doesn't happen before the update function unless the first
> update function completes during the operation of the second. This means
> that you can use them for a counter (say) without requiring your clients to
> know document _rev's, as in:
>
> http://stackoverflow.com/questions/2972068/couchdb-document-update-handlers-in-place-updates
>
> On Tue, Apr 5, 2011 at 7:19 PM, Sean Copenhaver
> <se...@gmail.com>wrote:
>
> > Keep in mind that besides both being doc db's they are very different
> > products.
> >
> > Also I'm not positive how far update functions can go. I believe the
> > version conflict happens before your update function.
> >
> > You should evaluate if you are trying to use couchdb too much like the
> > average relational db. Also think about if you can break up the docs in an
> > appropriate way to avoid the conflicts.
> >
> >
> >
> > On Apr 5, 2011, at 7:12 PM, Dennis Clark <db...@gmail.com> wrote:
> >
> > > What's your actual function? Update functions should work fine for your
> > use case, but can be a little finicky to get working, particularly the first
> > time.
> > >
> > > On Apr 5, 2011, at 7:01 PM, Luis Miguel Silva <
> > luismiguelferreirasilva@gmail.com> wrote:
> > >
> > > > Thank you! I'll probably have to evaluate it then...
> > > >
> > > > I've looked at the update handlers but i can't get them to work :o\...
> > > > I'm creating a _design document with a "updates" field with an update
> > > > function inside but i can't seem to get it to work.
> > > >
> > > > Either way, i think it's time i evaluate MongoDB :o((... (this was
> > > > unexpected, i was completely sold on CouchDB :o|).
> > > >
> > > > On Tue, Apr 5, 2011 at 4:50 PM, Sean Copenhaver
> > > > <se...@gmail.com> wrote:
> > > > > Ah, the problem is that couchdb does not do partial updates. It writes
> > the whole doc. MongoDB I believe does support partial updates though. No
> > experience with it.
> > > > >
> > > > >
> > > > >
> > > > > On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <
> > luismiguelferreirasilva@gmail.com> wrote:
> > > > >
> > > > > > More or less!
> > > > > >
> > > > > > The most common scenario will be:
> > > > > > - two or more processes writing to the same document, but only to a
> > > > > > specific attribute (not overwriting the whole document)
> > > > > >
> > > > > > If, by any chance, two processes overwrite the same field, i'm ok with
> > > > > > the last one always winning.
> > > > > >
> > > > > > Thanks,
> > > > > > Luis
> > > > > >
> > > > > > On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <
> > robert.newson@gmail.com> wrote:
> > > > > > > "Ideally, we would be able to update without specifying the _rev,
> > just
> > > > > > > posting (or, in this case PUTting) to the document..."
> > > > > > >
> > > > > > > So you want to blindly overwrite some unknown data?
> > > > > > >
> > > > > > > B.
> > > > > > >
> > > > > > > On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com>
> > wrote:
> > > > > > > > Luis,
> > > > > > > >
> > > > > > > > Checkout _update handlers:
> > > > > > > >
> > > > > > > > http://wiki.apache.org/couchdb/Document_Update_Handlers
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Zach
> > > > > > > >
> > > > > > > > On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
> > > > > > > > <lu...@gmail.com> wrote:
> > > > > > > > > Dear all,
> > > > > > > > >
> > > > > > > > > I'm trying to play around with updates and i'm bumping into some
> > problems.
> > > > > > > > >
> > > > > > > > > Let's image we have to clients that poll a document from the server
> > at
> > > > > > > > > the same time and get the same _rev.
> > > > > > > > > Then one of them updates the doc based on the _rev it got:
> > > > > > > > > [root@xkitten ~]# curl -X PUT -d
> > > > > > > > > '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> > > > > > > > > http://localhost:5984/benchmark/test?conflicts=true
> > > > > > > > > {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> > > > > > > > > Then another one tries to update the doc based on the same exact
> > _rev:
> > > > > > > > > [root@xkitten ~]# curl -X PUT -d
> > > > > > > > > '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> > > > > > > > > http://localhost:5984/benchmark/test?conflicts=true
> > > > > > > > > {"error":"conflict","reason":"Document update conflict."}
> > > > > > > > > [root@xkitten ~]#
> > > > > > > > >
> > > > > > > > > Is there a way to avoid this?! (like...make the update just create
> > a
> > > > > > > > > new _rev or something)??
> > > > > > > > >
> > > > > > > > > Ideally, we would be able to update without specifying the _rev,
> > just
> > > > > > > > > posting (or, in this case PUTting) to the document...
> > > > > > > > >
> > > > > > > > > Thoughts??
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > Luis
>
Re: Update conflicts?
Posted by Dennis Clark <db...@gmail.com>.
Actually no -- if you put to an update handler with an existing docid, the
handler gets a full copy of the then-current doc, _rev and all. As such, the
version conflict doesn't happen before the update function unless the first
update function completes during the operation of the second. This means
that you can use them for a counter (say) without requiring your clients to
know document _rev's, as in:
http://stackoverflow.com/questions/2972068/couchdb-document-update-handlers-in-place-updates
On Tue, Apr 5, 2011 at 7:19 PM, Sean Copenhaver
<se...@gmail.com>wrote:
> Keep in mind that besides both being doc db's they are very different
> products.
>
> Also I'm not positive how far update functions can go. I believe the
> version conflict happens before your update function.
>
> You should evaluate if you are trying to use couchdb too much like the
> average relational db. Also think about if you can break up the docs in an
> appropriate way to avoid the conflicts.
>
>
>
> On Apr 5, 2011, at 7:12 PM, Dennis Clark <db...@gmail.com> wrote:
>
> > What's your actual function? Update functions should work fine for your
> use case, but can be a little finicky to get working, particularly the first
> time.
> >
> > On Apr 5, 2011, at 7:01 PM, Luis Miguel Silva <
> luismiguelferreirasilva@gmail.com> wrote:
> >
> >> Thank you! I'll probably have to evaluate it then...
> >>
> >> I've looked at the update handlers but i can't get them to work :o\...
> >> I'm creating a _design document with a "updates" field with an update
> >> function inside but i can't seem to get it to work.
> >>
> >> Either way, i think it's time i evaluate MongoDB :o((... (this was
> >> unexpected, i was completely sold on CouchDB :o|).
> >>
> >> On Tue, Apr 5, 2011 at 4:50 PM, Sean Copenhaver
> >> <se...@gmail.com> wrote:
> >>> Ah, the problem is that couchdb does not do partial updates. It writes
> the whole doc. MongoDB I believe does support partial updates though. No
> experience with it.
> >>>
> >>>
> >>>
> >>> On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <
> luismiguelferreirasilva@gmail.com> wrote:
> >>>
> >>>> More or less!
> >>>>
> >>>> The most common scenario will be:
> >>>> - two or more processes writing to the same document, but only to a
> >>>> specific attribute (not overwriting the whole document)
> >>>>
> >>>> If, by any chance, two processes overwrite the same field, i'm ok with
> >>>> the last one always winning.
> >>>>
> >>>> Thanks,
> >>>> Luis
> >>>>
> >>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <
> robert.newson@gmail.com> wrote:
> >>>>> "Ideally, we would be able to update without specifying the _rev,
> just
> >>>>> posting (or, in this case PUTting) to the document..."
> >>>>>
> >>>>> So you want to blindly overwrite some unknown data?
> >>>>>
> >>>>> B.
> >>>>>
> >>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com>
> wrote:
> >>>>>> Luis,
> >>>>>>
> >>>>>> Checkout _update handlers:
> >>>>>>
> >>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
> >>>>>>
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Zach
> >>>>>>
> >>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
> >>>>>> <lu...@gmail.com> wrote:
> >>>>>>> Dear all,
> >>>>>>>
> >>>>>>> I'm trying to play around with updates and i'm bumping into some
> problems.
> >>>>>>>
> >>>>>>> Let's image we have to clients that poll a document from the server
> at
> >>>>>>> the same time and get the same _rev.
> >>>>>>> Then one of them updates the doc based on the _rev it got:
> >>>>>>> [root@xkitten ~]# curl -X PUT -d
> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
> >>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> >>>>>>> Then another one tries to update the doc based on the same exact
> _rev:
> >>>>>>> [root@xkitten ~]# curl -X PUT -d
> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
> >>>>>>> {"error":"conflict","reason":"Document update conflict."}
> >>>>>>> [root@xkitten ~]#
> >>>>>>>
> >>>>>>> Is there a way to avoid this?! (like...make the update just create
> a
> >>>>>>> new _rev or something)??
> >>>>>>>
> >>>>>>> Ideally, we would be able to update without specifying the _rev,
> just
> >>>>>>> posting (or, in this case PUTting) to the document...
> >>>>>>>
> >>>>>>> Thoughts??
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> Luis
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
>
Re: Update conflicts?
Posted by Sean Copenhaver <se...@gmail.com>.
Keep in mind that besides both being doc db's they are very different products.
Also I'm not positive how far update functions can go. I believe the version conflict happens before your update function.
You should evaluate if you are trying to use couchdb too much like the average relational db. Also think about if you can break up the docs in an appropriate way to avoid the conflicts.
On Apr 5, 2011, at 7:12 PM, Dennis Clark <db...@gmail.com> wrote:
> What's your actual function? Update functions should work fine for your use case, but can be a little finicky to get working, particularly the first time.
>
> On Apr 5, 2011, at 7:01 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
>
>> Thank you! I'll probably have to evaluate it then...
>>
>> I've looked at the update handlers but i can't get them to work :o\...
>> I'm creating a _design document with a "updates" field with an update
>> function inside but i can't seem to get it to work.
>>
>> Either way, i think it's time i evaluate MongoDB :o((... (this was
>> unexpected, i was completely sold on CouchDB :o|).
>>
>> On Tue, Apr 5, 2011 at 4:50 PM, Sean Copenhaver
>> <se...@gmail.com> wrote:
>>> Ah, the problem is that couchdb does not do partial updates. It writes the whole doc. MongoDB I believe does support partial updates though. No experience with it.
>>>
>>>
>>>
>>> On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
>>>
>>>> More or less!
>>>>
>>>> The most common scenario will be:
>>>> - two or more processes writing to the same document, but only to a
>>>> specific attribute (not overwriting the whole document)
>>>>
>>>> If, by any chance, two processes overwrite the same field, i'm ok with
>>>> the last one always winning.
>>>>
>>>> Thanks,
>>>> Luis
>>>>
>>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>>>>> "Ideally, we would be able to update without specifying the _rev, just
>>>>> posting (or, in this case PUTting) to the document..."
>>>>>
>>>>> So you want to blindly overwrite some unknown data?
>>>>>
>>>>> B.
>>>>>
>>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>>>>> Luis,
>>>>>>
>>>>>> Checkout _update handlers:
>>>>>>
>>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Zach
>>>>>>
>>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>>>>> <lu...@gmail.com> wrote:
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>>>>
>>>>>>> Let's image we have to clients that poll a document from the server at
>>>>>>> the same time and get the same _rev.
>>>>>>> Then one of them updates the doc based on the _rev it got:
>>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>>>>> Then another one tries to update the doc based on the same exact _rev:
>>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>>> {"error":"conflict","reason":"Document update conflict."}
>>>>>>> [root@xkitten ~]#
>>>>>>>
>>>>>>> Is there a way to avoid this?! (like...make the update just create a
>>>>>>> new _rev or something)??
>>>>>>>
>>>>>>> Ideally, we would be able to update without specifying the _rev, just
>>>>>>> posting (or, in this case PUTting) to the document...
>>>>>>>
>>>>>>> Thoughts??
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Luis
>>>>>>>
>>>>>>
>>>>>
>>>
Re: Update conflicts?
Posted by Dennis Clark <db...@gmail.com>.
What's your actual function? Update functions should work fine for your use case, but can be a little finicky to get working, particularly the first time.
On Apr 5, 2011, at 7:01 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
> Thank you! I'll probably have to evaluate it then...
>
> I've looked at the update handlers but i can't get them to work :o\...
> I'm creating a _design document with a "updates" field with an update
> function inside but i can't seem to get it to work.
>
> Either way, i think it's time i evaluate MongoDB :o((... (this was
> unexpected, i was completely sold on CouchDB :o|).
>
> On Tue, Apr 5, 2011 at 4:50 PM, Sean Copenhaver
> <se...@gmail.com> wrote:
>> Ah, the problem is that couchdb does not do partial updates. It writes the whole doc. MongoDB I believe does support partial updates though. No experience with it.
>>
>>
>>
>> On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
>>
>>> More or less!
>>>
>>> The most common scenario will be:
>>> - two or more processes writing to the same document, but only to a
>>> specific attribute (not overwriting the whole document)
>>>
>>> If, by any chance, two processes overwrite the same field, i'm ok with
>>> the last one always winning.
>>>
>>> Thanks,
>>> Luis
>>>
>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>>>> "Ideally, we would be able to update without specifying the _rev, just
>>>> posting (or, in this case PUTting) to the document..."
>>>>
>>>> So you want to blindly overwrite some unknown data?
>>>>
>>>> B.
>>>>
>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>>>> Luis,
>>>>>
>>>>> Checkout _update handlers:
>>>>>
>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Zach
>>>>>
>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>>>> <lu...@gmail.com> wrote:
>>>>>> Dear all,
>>>>>>
>>>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>>>
>>>>>> Let's image we have to clients that poll a document from the server at
>>>>>> the same time and get the same _rev.
>>>>>> Then one of them updates the doc based on the _rev it got:
>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>>>> Then another one tries to update the doc based on the same exact _rev:
>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>> {"error":"conflict","reason":"Document update conflict."}
>>>>>> [root@xkitten ~]#
>>>>>>
>>>>>> Is there a way to avoid this?! (like...make the update just create a
>>>>>> new _rev or something)??
>>>>>>
>>>>>> Ideally, we would be able to update without specifying the _rev, just
>>>>>> posting (or, in this case PUTting) to the document...
>>>>>>
>>>>>> Thoughts??
>>>>>>
>>>>>> Thank you,
>>>>>> Luis
>>>>>>
>>>>>
>>>>
>>
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
Thank you! I'll probably have to evaluate it then...
I've looked at the update handlers but i can't get them to work :o\...
I'm creating a _design document with a "updates" field with an update
function inside but i can't seem to get it to work.
Either way, i think it's time i evaluate MongoDB :o((... (this was
unexpected, i was completely sold on CouchDB :o|).
On Tue, Apr 5, 2011 at 4:50 PM, Sean Copenhaver
<se...@gmail.com> wrote:
> Ah, the problem is that couchdb does not do partial updates. It writes the whole doc. MongoDB I believe does support partial updates though. No experience with it.
>
>
>
> On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
>
>> More or less!
>>
>> The most common scenario will be:
>> - two or more processes writing to the same document, but only to a
>> specific attribute (not overwriting the whole document)
>>
>> If, by any chance, two processes overwrite the same field, i'm ok with
>> the last one always winning.
>>
>> Thanks,
>> Luis
>>
>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>>> "Ideally, we would be able to update without specifying the _rev, just
>>> posting (or, in this case PUTting) to the document..."
>>>
>>> So you want to blindly overwrite some unknown data?
>>>
>>> B.
>>>
>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>>> Luis,
>>>>
>>>> Checkout _update handlers:
>>>>
>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Zach
>>>>
>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>>> <lu...@gmail.com> wrote:
>>>>> Dear all,
>>>>>
>>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>>
>>>>> Let's image we have to clients that poll a document from the server at
>>>>> the same time and get the same _rev.
>>>>> Then one of them updates the doc based on the _rev it got:
>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>>> Then another one tries to update the doc based on the same exact _rev:
>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>> {"error":"conflict","reason":"Document update conflict."}
>>>>> [root@xkitten ~]#
>>>>>
>>>>> Is there a way to avoid this?! (like...make the update just create a
>>>>> new _rev or something)??
>>>>>
>>>>> Ideally, we would be able to update without specifying the _rev, just
>>>>> posting (or, in this case PUTting) to the document...
>>>>>
>>>>> Thoughts??
>>>>>
>>>>> Thank you,
>>>>> Luis
>>>>>
>>>>
>>>
>
Re: Update conflicts?
Posted by Sean Copenhaver <se...@gmail.com>.
Ah, the problem is that couchdb does not do partial updates. It writes the whole doc. MongoDB I believe does support partial updates though. No experience with it.
On Apr 5, 2011, at 6:41 PM, Luis Miguel Silva <lu...@gmail.com> wrote:
> More or less!
>
> The most common scenario will be:
> - two or more processes writing to the same document, but only to a
> specific attribute (not overwriting the whole document)
>
> If, by any chance, two processes overwrite the same field, i'm ok with
> the last one always winning.
>
> Thanks,
> Luis
>
> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>> "Ideally, we would be able to update without specifying the _rev, just
>> posting (or, in this case PUTting) to the document..."
>>
>> So you want to blindly overwrite some unknown data?
>>
>> B.
>>
>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>> Luis,
>>>
>>> Checkout _update handlers:
>>>
>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>
>>>
>>> Cheers,
>>>
>>> Zach
>>>
>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>> <lu...@gmail.com> wrote:
>>>> Dear all,
>>>>
>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>
>>>> Let's image we have to clients that poll a document from the server at
>>>> the same time and get the same _rev.
>>>> Then one of them updates the doc based on the _rev it got:
>>>> [root@xkitten ~]# curl -X PUT -d
>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>> Then another one tries to update the doc based on the same exact _rev:
>>>> [root@xkitten ~]# curl -X PUT -d
>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>> {"error":"conflict","reason":"Document update conflict."}
>>>> [root@xkitten ~]#
>>>>
>>>> Is there a way to avoid this?! (like...make the update just create a
>>>> new _rev or something)??
>>>>
>>>> Ideally, we would be able to update without specifying the _rev, just
>>>> posting (or, in this case PUTting) to the document...
>>>>
>>>> Thoughts??
>>>>
>>>> Thank you,
>>>> Luis
>>>>
>>>
>>
Re: Update conflicts?
Posted by Anup Bishnoi <pi...@gmail.com>.
Or just use a list function for the node_status view, if all you want is
that html response with all the information about one particular node.
Though you'll have to model the keys emitted from that view slightly
differently
On Wed, Apr 6, 2011 at 12:04 PM, Anup Bishnoi <pi...@gmail.com>wrote:
> you could join the different pieces of information about the node (which
> you get by one query on the view suggested above) on the page itself with
> javascript, instead of asking couch for everything embedded in an html
> response
>
>
> On Wed, Apr 6, 2011 at 11:53 AM, Luis Miguel Silva <
> luismiguelferreirasilva@gmail.com> wrote:
>
>> Sorry if my last email was too big :o).
>>
>> Well, one reason i wanted to avoid doing that is because it didn't
>> seem as easy to maintain as my original approach but i'll discuss your
>> suggestion with my team to see what they have to say.
>> Also, i just couldn't get join to work :o\...
>>
>> How would you create a view that joins data from those different types
>> of documents to create a single complete view of a node?
>> I've read the documentation on view joins but simply could not get it
>> to work :o\...
>>
>> Thank you,
>> Luis
>>
>> On Tue, Apr 5, 2011 at 9:12 PM, Ryan Ramage <ry...@gmail.com>
>> wrote:
>> > Luis,
>> >
>> > Thats a lot to take in, but a quick suggestion.
>> >
>> > Have a parent doc that looks like this:
>> > {
>> > id: node1,
>> > type: node,
>> > location: blah,
>> > }
>> >
>> > and some 'children' docs that look like this
>> >
>> > {
>> > id: 3232323323223-32323232322-3232,
>> > timestamp: 1299794532000,
>> > type: cpu,
>> > node: node1,
>> > cpu: 0.94,
>> > ccores: 4,
>> > acores: 4,
>> > cmemory: 4096,
>> > amemory: 1024
>> > }
>> >
>> > and
>> > {
>> > id: 3232323323223-32323232322-3232,
>> > timestamp: 1299794532000,
>> > type: disk,
>> > node: node1,
>> > disk: 100000
>> > }
>> > and
>> > {
>> > id: 433432323323223-3232323322332,
>> > timestamp: 1299794532000,
>> > type: netio,
>> > node: node1,
>> > in: 100,
>> > out: 200
>> > }
>> > and
>> > {
>> > id: 323432423432534534-534534-543534534
>> > timestamp: 1299794532000,
>> > type: generic,
>> > node: node1,
>> > name: "foo",
>> > value: "bar"
>> > }
>> >
>> > create a status view
>> > "node_status" : function (doc) {
>> > if (doc.type != 'node') {
>> > emit([doc.node, doc.type, doc.timestamp],null);
>> > }
>> > }
>> >
>> > This allows you to not have to ever update a doc. Just keep inserting.
>> > Couchdb is good at that.
>> >
>> >
>> >
>> >
>> > On Tue, Apr 5, 2011 at 7:20 PM, Luis Miguel Silva
>> > <lu...@gmail.com> wrote:
>> >> Thanks for your email Ryan.
>> >>
>> >> Let me give you some more information on what i'm trying to do...
>> >> Essentially, i have to create a "sort of CMDB" system that stores, not
>> only configuration data, but also operational data (so...i guess you could
>> call it a OMDB instead).
>> >>
>> >> Either way, my company develops a meta-scheduler that can be used for
>> HPC or Cloud environments. It will guarantee that your resources are used
>> the best way possible, maximizing their usage, based on the policies you set
>> up in it.
>> >>
>> >> To do that, our software needs to be aware of how the environment looks
>> and this is why an OMDB piece is very important for us (as it allows us to
>> store information on the environment).
>> >>
>> >> Also, our software talks with external resource managers by a protocol
>> we developed more than a dozen years ago called "WIKI" (not as in
>> "wikipedia" but, WIKI as in the hawayan word for fast). That protocol is
>> heavily based around key/value pairs so this is one of the reasons i was
>> EXTREMELY excited to find out that, with CouchDB's "view" functionality, i
>> would be able to map document attributes to more meaningful attributes that
>> our software understands (i.e. map the document's "available_cores"
>> attribute to "ccores" [the "consumable cores" parameter our software
>> understands]).
>> >>
>> >> Another important thing to notice is that resources can be off
>> different types: node (for bare metal nodes), vm (for vms running on nodes)
>> and storage (we can actually have more data types but those are enough to
>> exemplify what i'm talking about).
>> >>
>> >> This is why i created those "big documents" instead of smaller ones!
>> >> For instance, each document would represent an entire node (i.e. procs,
>> memory, etc).
>> >>
>> >> So my idea was to have an external process initially populate the
>> database with documents representing ALL the nodes we are managing (hence
>> why i started my benchmarks with 100K increments) and OTHER external
>> processes (i.e. other types of resource managers) would update individual
>> attributes in each document.
>> >>
>> >> Let's imagine a document with id "node01":
>> >> These fields would be updated by an agent that collected some of the
>> hardware specs:\
>> >> ccores: 4 // total cores on machine
>> >> acores: 4 // available cores on machine
>> >> cmemory: 4096 // total memory on machine
>> >> amemory: 1024 // available memory
>> >> cpuload: 94%
>> >> This field would be updated by our storage resource manager:
>> >> GMETRIC["disk"]: 1000000
>> >> And, for instance, these fields would be updated by a network resource
>> manager:
>> >> GMETRIC["NETIO"]: { "in":100, "out":200 }
>> >>
>> >> So, as you can see, different processes would manage the same document
>> (just different attributes in it).
>> >>
>> >> And the REALLY cool thing about the Views is the fact that our
>> customers could VERY easily adapt the database so that it would store THEIR
>> extra data and shove it in a generic parameter that our software woulder
>> understand [i.e. the GMETRIC parameters are generic metrics...).
>> >>
>> >> So, based on these requirements, do you have any suggestions on how we
>> should store our data (keeping its structure easy enough for external
>> consumers to maintain it without having to bust their heads figuring out the
>> logic behind the document attributes)?? :o)
>> >>
>> >> Thank you!
>> >> Luis Miguel Silva
>> >>
>> >> On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com> wrote:
>> >>
>> >>> Luis,
>> >>>
>> >>> Having the rev is very important when you update a doc. It lets you
>> >>> know that your piece of information is out of date. This is a good
>> >>> thing....
>> >>>
>> >>> I am wondering if the way you are modeling your data is not leading
>> >>> you to do this update with less chance of conflict. See if you can
>> >>> break your docs into even smaller docs. For example, I noticed from a
>> >>> prior post you had a lot of Arrays in your docs. If multiple processes
>> >>> are changing that array, you might be better served by making each
>> >>> element in the array a separate doc.
>> >>>
>> >>> Ryan
>> >>>
>> >>> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
>> >>> <lu...@gmail.com> wrote:
>> >>>> More or less!
>> >>>>
>> >>>> The most common scenario will be:
>> >>>> - two or more processes writing to the same document, but only to a
>> >>>> specific attribute (not overwriting the whole document)
>> >>>>
>> >>>> If, by any chance, two processes overwrite the same field, i'm ok
>> with
>> >>>> the last one always winning.
>> >>>>
>> >>>> Thanks,
>> >>>> Luis
>> >>>>
>> >>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <
>> robert.newson@gmail.com> wrote:
>> >>>>> "Ideally, we would be able to update without specifying the _rev,
>> just
>> >>>>> posting (or, in this case PUTting) to the document..."
>> >>>>>
>> >>>>> So you want to blindly overwrite some unknown data?
>> >>>>>
>> >>>>> B.
>> >>>>>
>> >>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com>
>> wrote:
>> >>>>>> Luis,
>> >>>>>>
>> >>>>>> Checkout _update handlers:
>> >>>>>>
>> >>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>> >>>>>>
>> >>>>>>
>> >>>>>> Cheers,
>> >>>>>>
>> >>>>>> Zach
>> >>>>>>
>> >>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>> >>>>>> <lu...@gmail.com> wrote:
>> >>>>>>> Dear all,
>> >>>>>>>
>> >>>>>>> I'm trying to play around with updates and i'm bumping into some
>> problems.
>> >>>>>>>
>> >>>>>>> Let's image we have to clients that poll a document from the
>> server at
>> >>>>>>> the same time and get the same _rev.
>> >>>>>>> Then one of them updates the doc based on the _rev it got:
>> >>>>>>> [root@xkitten ~]# curl -X PUT -d
>> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>> >>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>> >>>>>>> Then another one tries to update the doc based on the same exact
>> _rev:
>> >>>>>>> [root@xkitten ~]# curl -X PUT -d
>> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>> >>>>>>> {"error":"conflict","reason":"Document update conflict."}
>> >>>>>>> [root@xkitten ~]#
>> >>>>>>>
>> >>>>>>> Is there a way to avoid this?! (like...make the update just create
>> a
>> >>>>>>> new _rev or something)??
>> >>>>>>>
>> >>>>>>> Ideally, we would be able to update without specifying the _rev,
>> just
>> >>>>>>> posting (or, in this case PUTting) to the document...
>> >>>>>>>
>> >>>>>>> Thoughts??
>> >>>>>>>
>> >>>>>>> Thank you,
>> >>>>>>> Luis
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> >
>>
>
>
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
But is there no way to do it server side? :o)
That would be SOOOO much better as i want to maintain a "single view
of the database" (so that everybody querying the same view gets the
same results).
Plus, your approach doesn't allow me to specify my own attribute names
(does it??):
i.e.
emit(doc._id,
{
node: doc._id,
STATE: doc.secondary_state,
OS: doc.oslist,
ALIAS: doc.alias,
FEATURE: doc.vlans,
"GMETRIC[numvms]": doc.numvms,
NETADDR: doc.netaddress,
VARATTR: { "HVTYPE":doc.hvtype},
VARIABLE: doc.variables,
OSLIST: doc.oslist,
VMOSLIST: doc.vmoslist,
}
);
Like i mentioned in a previous document, that is a HUGE deal to us
because the attributes themselves have no meaning to the consumers. So
that is why it is EXTREMELY important for us to shape the information
in a meaningful way on the server side!
p.s. thank you so much for your help.
On Wed, Apr 6, 2011 at 12:56 AM, Anup Bishnoi <pi...@gmail.com> wrote:
> you've already got the answer here
>
> On Wed, Apr 6, 2011 at 12:19 PM, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>>
>> Yeah but the above view generates different documents:
>> {"total_rows":4,"offset":0,"rows":[
>>
>> {"id":"92fe8c96f90e21d68a414bbd1700f3d7","key":["node01","cpu",1299794532000,0.94],"value":null},
>>
>> {"id":"92fe8c96f90e21d68a414bbd1700ffee","key":["node01","disk",1299794532000,null],"value":null},
>>
>> {"id":"92fe8c96f90e21d68a414bbd1701180e","key":["node01","generic",1299794532000,null],"value":null},
>>
>> {"id":"92fe8c96f90e21d68a414bbd170109ce","key":["node01","netio",1299794532000,null],"value":null}
>> ]}
>>
> i'm assuming you're making this view query with ajax and you get these
> results.
> now all you need to do is walk through these response items with client side
> js and build the one doc you need! all the pieces required to build the doc
> are already there with you in your client side js
> i'll be happy to keep answering, lets get this solved
>
>>
>> Any way i can return ONE single doc per full result?
>> i.e. something like:
>> {"total_rows":1,"offset":0,"rows":[
>>
>> {"id":"node01","key":"node01","value":{"node":"node01","STATE":"Unknown:sshd","ALIAS":"node01","FEATURE":"[vlan611]","GMETRIC[numvms]":13,"NETADDR":"10.40.130.146","VARATTR":{"HVTYPE":"esx"},"VARIABLE":[{"provision_status":2},{"another_variable":"something"}]}},
>> ]}
>>
>> (or, in other words, joining all the fields from the different
>> documents in one single doc)??
>>
>> On Wed, Apr 6, 2011 at 12:34 AM, Anup Bishnoi <pi...@gmail.com>
>> wrote:
>> > you could join the different pieces of information about the node (which
>> > you
>> > get by one query on the view suggested above) on the page itself with
>> > javascript, instead of asking couch for everything embedded in an html
>> > response
>> >
>> > On Wed, Apr 6, 2011 at 11:53 AM, Luis Miguel Silva
>> > <lu...@gmail.com> wrote:
>> >>
>> >> Sorry if my last email was too big :o).
>> >>
>> >> Well, one reason i wanted to avoid doing that is because it didn't
>> >> seem as easy to maintain as my original approach but i'll discuss your
>> >> suggestion with my team to see what they have to say.
>> >> Also, i just couldn't get join to work :o\...
>> >>
>> >> How would you create a view that joins data from those different types
>> >> of documents to create a single complete view of a node?
>> >> I've read the documentation on view joins but simply could not get it
>> >> to work :o\...
>> >>
>> >> Thank you,
>> >> Luis
>> >>
>> >> On Tue, Apr 5, 2011 at 9:12 PM, Ryan Ramage <ry...@gmail.com>
>> >> wrote:
>> >> > Luis,
>> >> >
>> >> > Thats a lot to take in, but a quick suggestion.
>> >> >
>> >> > Have a parent doc that looks like this:
>> >> > {
>> >> > id: node1,
>> >> > type: node,
>> >> > location: blah,
>> >> > }
>> >> >
>> >> > and some 'children' docs that look like this
>> >> >
>> >> > {
>> >> > id: 3232323323223-32323232322-3232,
>> >> > timestamp: 1299794532000,
>> >> > type: cpu,
>> >> > node: node1,
>> >> > cpu: 0.94,
>> >> > ccores: 4,
>> >> > acores: 4,
>> >> > cmemory: 4096,
>> >> > amemory: 1024
>> >> > }
>> >> >
>> >> > and
>> >> > {
>> >> > id: 3232323323223-32323232322-3232,
>> >> > timestamp: 1299794532000,
>> >> > type: disk,
>> >> > node: node1,
>> >> > disk: 100000
>> >> > }
>> >> > and
>> >> > {
>> >> > id: 433432323323223-3232323322332,
>> >> > timestamp: 1299794532000,
>> >> > type: netio,
>> >> > node: node1,
>> >> > in: 100,
>> >> > out: 200
>> >> > }
>> >> > and
>> >> > {
>> >> > id: 323432423432534534-534534-543534534
>> >> > timestamp: 1299794532000,
>> >> > type: generic,
>> >> > node: node1,
>> >> > name: "foo",
>> >> > value: "bar"
>> >> > }
>> >> >
>> >> > create a status view
>> >> > "node_status" : function (doc) {
>> >> > if (doc.type != 'node') {
>> >> > emit([doc.node, doc.type, doc.timestamp],null);
>> >> > }
>> >> > }
>> >> >
>> >> > This allows you to not have to ever update a doc. Just keep
>> >> > inserting.
>> >> > Couchdb is good at that.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Apr 5, 2011 at 7:20 PM, Luis Miguel Silva
>> >> > <lu...@gmail.com> wrote:
>> >> >> Thanks for your email Ryan.
>> >> >>
>> >> >> Let me give you some more information on what i'm trying to do...
>> >> >> Essentially, i have to create a "sort of CMDB" system that stores,
>> >> >> not
>> >> >> only configuration data, but also operational data (so...i guess you
>> >> >> could
>> >> >> call it a OMDB instead).
>> >> >>
>> >> >> Either way, my company develops a meta-scheduler that can be used
>> >> >> for
>> >> >> HPC or Cloud environments. It will guarantee that your resources are
>> >> >> used
>> >> >> the best way possible, maximizing their usage, based on the policies
>> >> >> you set
>> >> >> up in it.
>> >> >>
>> >> >> To do that, our software needs to be aware of how the environment
>> >> >> looks
>> >> >> and this is why an OMDB piece is very important for us (as it allows
>> >> >> us to
>> >> >> store information on the environment).
>> >> >>
>> >> >> Also, our software talks with external resource managers by a
>> >> >> protocol
>> >> >> we developed more than a dozen years ago called "WIKI" (not as in
>> >> >> "wikipedia" but, WIKI as in the hawayan word for fast). That
>> >> >> protocol is
>> >> >> heavily based around key/value pairs so this is one of the reasons i
>> >> >> was
>> >> >> EXTREMELY excited to find out that, with CouchDB's "view"
>> >> >> functionality, i
>> >> >> would be able to map document attributes to more meaningful
>> >> >> attributes that
>> >> >> our software understands (i.e. map the document's "available_cores"
>> >> >> attribute to "ccores" [the "consumable cores" parameter our software
>> >> >> understands]).
>> >> >>
>> >> >> Another important thing to notice is that resources can be off
>> >> >> different types: node (for bare metal nodes), vm (for vms running on
>> >> >> nodes)
>> >> >> and storage (we can actually have more data types but those are
>> >> >> enough to
>> >> >> exemplify what i'm talking about).
>> >> >>
>> >> >> This is why i created those "big documents" instead of smaller ones!
>> >> >> For instance, each document would represent an entire node (i.e.
>> >> >> procs,
>> >> >> memory, etc).
>> >> >>
>> >> >> So my idea was to have an external process initially populate the
>> >> >> database with documents representing ALL the nodes we are managing
>> >> >> (hence
>> >> >> why i started my benchmarks with 100K increments) and OTHER external
>> >> >> processes (i.e. other types of resource managers) would update
>> >> >> individual
>> >> >> attributes in each document.
>> >> >>
>> >> >> Let's imagine a document with id "node01":
>> >> >> These fields would be updated by an agent that collected some of the
>> >> >> hardware specs:\
>> >> >> ccores: 4 // total cores on machine
>> >> >> acores: 4 // available cores on machine
>> >> >> cmemory: 4096 // total memory on machine
>> >> >> amemory: 1024 // available memory
>> >> >> cpuload: 94%
>> >> >> This field would be updated by our storage resource manager:
>> >> >> GMETRIC["disk"]: 1000000
>> >> >> And, for instance, these fields would be updated by a network
>> >> >> resource
>> >> >> manager:
>> >> >> GMETRIC["NETIO"]: { "in":100, "out":200 }
>> >> >>
>> >> >> So, as you can see, different processes would manage the same
>> >> >> document
>> >> >> (just different attributes in it).
>> >> >>
>> >> >> And the REALLY cool thing about the Views is the fact that our
>> >> >> customers could VERY easily adapt the database so that it would
>> >> >> store THEIR
>> >> >> extra data and shove it in a generic parameter that our software
>> >> >> woulder
>> >> >> understand [i.e. the GMETRIC parameters are generic metrics...).
>> >> >>
>> >> >> So, based on these requirements, do you have any suggestions on how
>> >> >> we
>> >> >> should store our data (keeping its structure easy enough for
>> >> >> external
>> >> >> consumers to maintain it without having to bust their heads figuring
>> >> >> out the
>> >> >> logic behind the document attributes)?? :o)
>> >> >>
>> >> >> Thank you!
>> >> >> Luis Miguel Silva
>> >> >>
>> >> >> On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >>> Luis,
>> >> >>>
>> >> >>> Having the rev is very important when you update a doc. It lets you
>> >> >>> know that your piece of information is out of date. This is a good
>> >> >>> thing....
>> >> >>>
>> >> >>> I am wondering if the way you are modeling your data is not leading
>> >> >>> you to do this update with less chance of conflict. See if you can
>> >> >>> break your docs into even smaller docs. For example, I noticed from
>> >> >>> a
>> >> >>> prior post you had a lot of Arrays in your docs. If multiple
>> >> >>> processes
>> >> >>> are changing that array, you might be better served by making each
>> >> >>> element in the array a separate doc.
>> >> >>>
>> >> >>> Ryan
>> >> >>>
>> >> >>> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
>> >> >>> <lu...@gmail.com> wrote:
>> >> >>>> More or less!
>> >> >>>>
>> >> >>>> The most common scenario will be:
>> >> >>>> - two or more processes writing to the same document, but only to
>> >> >>>> a
>> >> >>>> specific attribute (not overwriting the whole document)
>> >> >>>>
>> >> >>>> If, by any chance, two processes overwrite the same field, i'm ok
>> >> >>>> with
>> >> >>>> the last one always winning.
>> >> >>>>
>> >> >>>> Thanks,
>> >> >>>> Luis
>> >> >>>>
>> >> >>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson
>> >> >>>> <ro...@gmail.com> wrote:
>> >> >>>>> "Ideally, we would be able to update without specifying the _rev,
>> >> >>>>> just
>> >> >>>>> posting (or, in this case PUTting) to the document..."
>> >> >>>>>
>> >> >>>>> So you want to blindly overwrite some unknown data?
>> >> >>>>>
>> >> >>>>> B.
>> >> >>>>>
>> >> >>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com>
>> >> >>>>> wrote:
>> >> >>>>>> Luis,
>> >> >>>>>>
>> >> >>>>>> Checkout _update handlers:
>> >> >>>>>>
>> >> >>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> Cheers,
>> >> >>>>>>
>> >> >>>>>> Zach
>> >> >>>>>>
>> >> >>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>> >> >>>>>> <lu...@gmail.com> wrote:
>> >> >>>>>>> Dear all,
>> >> >>>>>>>
>> >> >>>>>>> I'm trying to play around with updates and i'm bumping into
>> >> >>>>>>> some
>> >> >>>>>>> problems.
>> >> >>>>>>>
>> >> >>>>>>> Let's image we have to clients that poll a document from the
>> >> >>>>>>> server at
>> >> >>>>>>> the same time and get the same _rev.
>> >> >>>>>>> Then one of them updates the doc based on the _rev it got:
>> >> >>>>>>> [root@xkitten ~]# curl -X PUT -d
>> >> >>>>>>>
>> >> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>> >> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>> >> >>>>>>>
>> >> >>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>> >> >>>>>>> Then another one tries to update the doc based on the same
>> >> >>>>>>> exact
>> >> >>>>>>> _rev:
>> >> >>>>>>> [root@xkitten ~]# curl -X PUT -d
>> >> >>>>>>>
>> >> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>> >> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>> >> >>>>>>> {"error":"conflict","reason":"Document update conflict."}
>> >> >>>>>>> [root@xkitten ~]#
>> >> >>>>>>>
>> >> >>>>>>> Is there a way to avoid this?! (like...make the update just
>> >> >>>>>>> create
>> >> >>>>>>> a
>> >> >>>>>>> new _rev or something)??
>> >> >>>>>>>
>> >> >>>>>>> Ideally, we would be able to update without specifying the
>> >> >>>>>>> _rev,
>> >> >>>>>>> just
>> >> >>>>>>> posting (or, in this case PUTting) to the document...
>> >> >>>>>>>
>> >> >>>>>>> Thoughts??
>> >> >>>>>>>
>> >> >>>>>>> Thank you,
>> >> >>>>>>> Luis
>> >> >>>>>>>
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>
>> >> >
>> >
>> >
>
>
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
Yeah but the above view generates different documents:
{"total_rows":4,"offset":0,"rows":[
{"id":"92fe8c96f90e21d68a414bbd1700f3d7","key":["node01","cpu",1299794532000,0.94],"value":null},
{"id":"92fe8c96f90e21d68a414bbd1700ffee","key":["node01","disk",1299794532000,null],"value":null},
{"id":"92fe8c96f90e21d68a414bbd1701180e","key":["node01","generic",1299794532000,null],"value":null},
{"id":"92fe8c96f90e21d68a414bbd170109ce","key":["node01","netio",1299794532000,null],"value":null}
]}
Any way i can return ONE single doc per full result?
i.e. something like:
{"total_rows":1,"offset":0,"rows":[
{"id":"node01","key":"node01","value":{"node":"node01","STATE":"Unknown:sshd","ALIAS":"node01","FEATURE":"[vlan611]","GMETRIC[numvms]":13,"NETADDR":"10.40.130.146","VARATTR":{"HVTYPE":"esx"},"VARIABLE":[{"provision_status":2},{"another_variable":"something"}]}},
]}
(or, in other words, joining all the fields from the different
documents in one single doc)??
On Wed, Apr 6, 2011 at 12:34 AM, Anup Bishnoi <pi...@gmail.com> wrote:
> you could join the different pieces of information about the node (which you
> get by one query on the view suggested above) on the page itself with
> javascript, instead of asking couch for everything embedded in an html
> response
>
> On Wed, Apr 6, 2011 at 11:53 AM, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>>
>> Sorry if my last email was too big :o).
>>
>> Well, one reason i wanted to avoid doing that is because it didn't
>> seem as easy to maintain as my original approach but i'll discuss your
>> suggestion with my team to see what they have to say.
>> Also, i just couldn't get join to work :o\...
>>
>> How would you create a view that joins data from those different types
>> of documents to create a single complete view of a node?
>> I've read the documentation on view joins but simply could not get it
>> to work :o\...
>>
>> Thank you,
>> Luis
>>
>> On Tue, Apr 5, 2011 at 9:12 PM, Ryan Ramage <ry...@gmail.com> wrote:
>> > Luis,
>> >
>> > Thats a lot to take in, but a quick suggestion.
>> >
>> > Have a parent doc that looks like this:
>> > {
>> > id: node1,
>> > type: node,
>> > location: blah,
>> > }
>> >
>> > and some 'children' docs that look like this
>> >
>> > {
>> > id: 3232323323223-32323232322-3232,
>> > timestamp: 1299794532000,
>> > type: cpu,
>> > node: node1,
>> > cpu: 0.94,
>> > ccores: 4,
>> > acores: 4,
>> > cmemory: 4096,
>> > amemory: 1024
>> > }
>> >
>> > and
>> > {
>> > id: 3232323323223-32323232322-3232,
>> > timestamp: 1299794532000,
>> > type: disk,
>> > node: node1,
>> > disk: 100000
>> > }
>> > and
>> > {
>> > id: 433432323323223-3232323322332,
>> > timestamp: 1299794532000,
>> > type: netio,
>> > node: node1,
>> > in: 100,
>> > out: 200
>> > }
>> > and
>> > {
>> > id: 323432423432534534-534534-543534534
>> > timestamp: 1299794532000,
>> > type: generic,
>> > node: node1,
>> > name: "foo",
>> > value: "bar"
>> > }
>> >
>> > create a status view
>> > "node_status" : function (doc) {
>> > if (doc.type != 'node') {
>> > emit([doc.node, doc.type, doc.timestamp],null);
>> > }
>> > }
>> >
>> > This allows you to not have to ever update a doc. Just keep inserting.
>> > Couchdb is good at that.
>> >
>> >
>> >
>> >
>> > On Tue, Apr 5, 2011 at 7:20 PM, Luis Miguel Silva
>> > <lu...@gmail.com> wrote:
>> >> Thanks for your email Ryan.
>> >>
>> >> Let me give you some more information on what i'm trying to do...
>> >> Essentially, i have to create a "sort of CMDB" system that stores, not
>> >> only configuration data, but also operational data (so...i guess you could
>> >> call it a OMDB instead).
>> >>
>> >> Either way, my company develops a meta-scheduler that can be used for
>> >> HPC or Cloud environments. It will guarantee that your resources are used
>> >> the best way possible, maximizing their usage, based on the policies you set
>> >> up in it.
>> >>
>> >> To do that, our software needs to be aware of how the environment looks
>> >> and this is why an OMDB piece is very important for us (as it allows us to
>> >> store information on the environment).
>> >>
>> >> Also, our software talks with external resource managers by a protocol
>> >> we developed more than a dozen years ago called "WIKI" (not as in
>> >> "wikipedia" but, WIKI as in the hawayan word for fast). That protocol is
>> >> heavily based around key/value pairs so this is one of the reasons i was
>> >> EXTREMELY excited to find out that, with CouchDB's "view" functionality, i
>> >> would be able to map document attributes to more meaningful attributes that
>> >> our software understands (i.e. map the document's "available_cores"
>> >> attribute to "ccores" [the "consumable cores" parameter our software
>> >> understands]).
>> >>
>> >> Another important thing to notice is that resources can be off
>> >> different types: node (for bare metal nodes), vm (for vms running on nodes)
>> >> and storage (we can actually have more data types but those are enough to
>> >> exemplify what i'm talking about).
>> >>
>> >> This is why i created those "big documents" instead of smaller ones!
>> >> For instance, each document would represent an entire node (i.e. procs,
>> >> memory, etc).
>> >>
>> >> So my idea was to have an external process initially populate the
>> >> database with documents representing ALL the nodes we are managing (hence
>> >> why i started my benchmarks with 100K increments) and OTHER external
>> >> processes (i.e. other types of resource managers) would update individual
>> >> attributes in each document.
>> >>
>> >> Let's imagine a document with id "node01":
>> >> These fields would be updated by an agent that collected some of the
>> >> hardware specs:\
>> >> ccores: 4 // total cores on machine
>> >> acores: 4 // available cores on machine
>> >> cmemory: 4096 // total memory on machine
>> >> amemory: 1024 // available memory
>> >> cpuload: 94%
>> >> This field would be updated by our storage resource manager:
>> >> GMETRIC["disk"]: 1000000
>> >> And, for instance, these fields would be updated by a network resource
>> >> manager:
>> >> GMETRIC["NETIO"]: { "in":100, "out":200 }
>> >>
>> >> So, as you can see, different processes would manage the same document
>> >> (just different attributes in it).
>> >>
>> >> And the REALLY cool thing about the Views is the fact that our
>> >> customers could VERY easily adapt the database so that it would store THEIR
>> >> extra data and shove it in a generic parameter that our software woulder
>> >> understand [i.e. the GMETRIC parameters are generic metrics...).
>> >>
>> >> So, based on these requirements, do you have any suggestions on how we
>> >> should store our data (keeping its structure easy enough for external
>> >> consumers to maintain it without having to bust their heads figuring out the
>> >> logic behind the document attributes)?? :o)
>> >>
>> >> Thank you!
>> >> Luis Miguel Silva
>> >>
>> >> On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com> wrote:
>> >>
>> >>> Luis,
>> >>>
>> >>> Having the rev is very important when you update a doc. It lets you
>> >>> know that your piece of information is out of date. This is a good
>> >>> thing....
>> >>>
>> >>> I am wondering if the way you are modeling your data is not leading
>> >>> you to do this update with less chance of conflict. See if you can
>> >>> break your docs into even smaller docs. For example, I noticed from a
>> >>> prior post you had a lot of Arrays in your docs. If multiple processes
>> >>> are changing that array, you might be better served by making each
>> >>> element in the array a separate doc.
>> >>>
>> >>> Ryan
>> >>>
>> >>> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
>> >>> <lu...@gmail.com> wrote:
>> >>>> More or less!
>> >>>>
>> >>>> The most common scenario will be:
>> >>>> - two or more processes writing to the same document, but only to a
>> >>>> specific attribute (not overwriting the whole document)
>> >>>>
>> >>>> If, by any chance, two processes overwrite the same field, i'm ok
>> >>>> with
>> >>>> the last one always winning.
>> >>>>
>> >>>> Thanks,
>> >>>> Luis
>> >>>>
>> >>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson
>> >>>> <ro...@gmail.com> wrote:
>> >>>>> "Ideally, we would be able to update without specifying the _rev,
>> >>>>> just
>> >>>>> posting (or, in this case PUTting) to the document..."
>> >>>>>
>> >>>>> So you want to blindly overwrite some unknown data?
>> >>>>>
>> >>>>> B.
>> >>>>>
>> >>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com>
>> >>>>> wrote:
>> >>>>>> Luis,
>> >>>>>>
>> >>>>>> Checkout _update handlers:
>> >>>>>>
>> >>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>> >>>>>>
>> >>>>>>
>> >>>>>> Cheers,
>> >>>>>>
>> >>>>>> Zach
>> >>>>>>
>> >>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>> >>>>>> <lu...@gmail.com> wrote:
>> >>>>>>> Dear all,
>> >>>>>>>
>> >>>>>>> I'm trying to play around with updates and i'm bumping into some
>> >>>>>>> problems.
>> >>>>>>>
>> >>>>>>> Let's image we have to clients that poll a document from the
>> >>>>>>> server at
>> >>>>>>> the same time and get the same _rev.
>> >>>>>>> Then one of them updates the doc based on the _rev it got:
>> >>>>>>> [root@xkitten ~]# curl -X PUT -d
>> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>> >>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>> >>>>>>> Then another one tries to update the doc based on the same exact
>> >>>>>>> _rev:
>> >>>>>>> [root@xkitten ~]# curl -X PUT -d
>> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>> >>>>>>> {"error":"conflict","reason":"Document update conflict."}
>> >>>>>>> [root@xkitten ~]#
>> >>>>>>>
>> >>>>>>> Is there a way to avoid this?! (like...make the update just create
>> >>>>>>> a
>> >>>>>>> new _rev or something)??
>> >>>>>>>
>> >>>>>>> Ideally, we would be able to update without specifying the _rev,
>> >>>>>>> just
>> >>>>>>> posting (or, in this case PUTting) to the document...
>> >>>>>>>
>> >>>>>>> Thoughts??
>> >>>>>>>
>> >>>>>>> Thank you,
>> >>>>>>> Luis
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> >
>
>
Re: Update conflicts?
Posted by Anup Bishnoi <pi...@gmail.com>.
you could join the different pieces of information about the node (which you
get by one query on the view suggested above) on the page itself with
javascript, instead of asking couch for everything embedded in an html
response
On Wed, Apr 6, 2011 at 11:53 AM, Luis Miguel Silva <
luismiguelferreirasilva@gmail.com> wrote:
> Sorry if my last email was too big :o).
>
> Well, one reason i wanted to avoid doing that is because it didn't
> seem as easy to maintain as my original approach but i'll discuss your
> suggestion with my team to see what they have to say.
> Also, i just couldn't get join to work :o\...
>
> How would you create a view that joins data from those different types
> of documents to create a single complete view of a node?
> I've read the documentation on view joins but simply could not get it
> to work :o\...
>
> Thank you,
> Luis
>
> On Tue, Apr 5, 2011 at 9:12 PM, Ryan Ramage <ry...@gmail.com> wrote:
> > Luis,
> >
> > Thats a lot to take in, but a quick suggestion.
> >
> > Have a parent doc that looks like this:
> > {
> > id: node1,
> > type: node,
> > location: blah,
> > }
> >
> > and some 'children' docs that look like this
> >
> > {
> > id: 3232323323223-32323232322-3232,
> > timestamp: 1299794532000,
> > type: cpu,
> > node: node1,
> > cpu: 0.94,
> > ccores: 4,
> > acores: 4,
> > cmemory: 4096,
> > amemory: 1024
> > }
> >
> > and
> > {
> > id: 3232323323223-32323232322-3232,
> > timestamp: 1299794532000,
> > type: disk,
> > node: node1,
> > disk: 100000
> > }
> > and
> > {
> > id: 433432323323223-3232323322332,
> > timestamp: 1299794532000,
> > type: netio,
> > node: node1,
> > in: 100,
> > out: 200
> > }
> > and
> > {
> > id: 323432423432534534-534534-543534534
> > timestamp: 1299794532000,
> > type: generic,
> > node: node1,
> > name: "foo",
> > value: "bar"
> > }
> >
> > create a status view
> > "node_status" : function (doc) {
> > if (doc.type != 'node') {
> > emit([doc.node, doc.type, doc.timestamp],null);
> > }
> > }
> >
> > This allows you to not have to ever update a doc. Just keep inserting.
> > Couchdb is good at that.
> >
> >
> >
> >
> > On Tue, Apr 5, 2011 at 7:20 PM, Luis Miguel Silva
> > <lu...@gmail.com> wrote:
> >> Thanks for your email Ryan.
> >>
> >> Let me give you some more information on what i'm trying to do...
> >> Essentially, i have to create a "sort of CMDB" system that stores, not
> only configuration data, but also operational data (so...i guess you could
> call it a OMDB instead).
> >>
> >> Either way, my company develops a meta-scheduler that can be used for
> HPC or Cloud environments. It will guarantee that your resources are used
> the best way possible, maximizing their usage, based on the policies you set
> up in it.
> >>
> >> To do that, our software needs to be aware of how the environment looks
> and this is why an OMDB piece is very important for us (as it allows us to
> store information on the environment).
> >>
> >> Also, our software talks with external resource managers by a protocol
> we developed more than a dozen years ago called "WIKI" (not as in
> "wikipedia" but, WIKI as in the hawayan word for fast). That protocol is
> heavily based around key/value pairs so this is one of the reasons i was
> EXTREMELY excited to find out that, with CouchDB's "view" functionality, i
> would be able to map document attributes to more meaningful attributes that
> our software understands (i.e. map the document's "available_cores"
> attribute to "ccores" [the "consumable cores" parameter our software
> understands]).
> >>
> >> Another important thing to notice is that resources can be off different
> types: node (for bare metal nodes), vm (for vms running on nodes) and
> storage (we can actually have more data types but those are enough to
> exemplify what i'm talking about).
> >>
> >> This is why i created those "big documents" instead of smaller ones!
> >> For instance, each document would represent an entire node (i.e. procs,
> memory, etc).
> >>
> >> So my idea was to have an external process initially populate the
> database with documents representing ALL the nodes we are managing (hence
> why i started my benchmarks with 100K increments) and OTHER external
> processes (i.e. other types of resource managers) would update individual
> attributes in each document.
> >>
> >> Let's imagine a document with id "node01":
> >> These fields would be updated by an agent that collected some of the
> hardware specs:\
> >> ccores: 4 // total cores on machine
> >> acores: 4 // available cores on machine
> >> cmemory: 4096 // total memory on machine
> >> amemory: 1024 // available memory
> >> cpuload: 94%
> >> This field would be updated by our storage resource manager:
> >> GMETRIC["disk"]: 1000000
> >> And, for instance, these fields would be updated by a network resource
> manager:
> >> GMETRIC["NETIO"]: { "in":100, "out":200 }
> >>
> >> So, as you can see, different processes would manage the same document
> (just different attributes in it).
> >>
> >> And the REALLY cool thing about the Views is the fact that our customers
> could VERY easily adapt the database so that it would store THEIR extra data
> and shove it in a generic parameter that our software woulder understand
> [i.e. the GMETRIC parameters are generic metrics...).
> >>
> >> So, based on these requirements, do you have any suggestions on how we
> should store our data (keeping its structure easy enough for external
> consumers to maintain it without having to bust their heads figuring out the
> logic behind the document attributes)?? :o)
> >>
> >> Thank you!
> >> Luis Miguel Silva
> >>
> >> On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com> wrote:
> >>
> >>> Luis,
> >>>
> >>> Having the rev is very important when you update a doc. It lets you
> >>> know that your piece of information is out of date. This is a good
> >>> thing....
> >>>
> >>> I am wondering if the way you are modeling your data is not leading
> >>> you to do this update with less chance of conflict. See if you can
> >>> break your docs into even smaller docs. For example, I noticed from a
> >>> prior post you had a lot of Arrays in your docs. If multiple processes
> >>> are changing that array, you might be better served by making each
> >>> element in the array a separate doc.
> >>>
> >>> Ryan
> >>>
> >>> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
> >>> <lu...@gmail.com> wrote:
> >>>> More or less!
> >>>>
> >>>> The most common scenario will be:
> >>>> - two or more processes writing to the same document, but only to a
> >>>> specific attribute (not overwriting the whole document)
> >>>>
> >>>> If, by any chance, two processes overwrite the same field, i'm ok with
> >>>> the last one always winning.
> >>>>
> >>>> Thanks,
> >>>> Luis
> >>>>
> >>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <
> robert.newson@gmail.com> wrote:
> >>>>> "Ideally, we would be able to update without specifying the _rev,
> just
> >>>>> posting (or, in this case PUTting) to the document..."
> >>>>>
> >>>>> So you want to blindly overwrite some unknown data?
> >>>>>
> >>>>> B.
> >>>>>
> >>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com>
> wrote:
> >>>>>> Luis,
> >>>>>>
> >>>>>> Checkout _update handlers:
> >>>>>>
> >>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
> >>>>>>
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Zach
> >>>>>>
> >>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
> >>>>>> <lu...@gmail.com> wrote:
> >>>>>>> Dear all,
> >>>>>>>
> >>>>>>> I'm trying to play around with updates and i'm bumping into some
> problems.
> >>>>>>>
> >>>>>>> Let's image we have to clients that poll a document from the server
> at
> >>>>>>> the same time and get the same _rev.
> >>>>>>> Then one of them updates the doc based on the _rev it got:
> >>>>>>> [root@xkitten ~]# curl -X PUT -d
> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
> >>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> >>>>>>> Then another one tries to update the doc based on the same exact
> _rev:
> >>>>>>> [root@xkitten ~]# curl -X PUT -d
> >>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> >>>>>>> http://localhost:5984/benchmark/test?conflicts=true
> >>>>>>> {"error":"conflict","reason":"Document update conflict."}
> >>>>>>> [root@xkitten ~]#
> >>>>>>>
> >>>>>>> Is there a way to avoid this?! (like...make the update just create
> a
> >>>>>>> new _rev or something)??
> >>>>>>>
> >>>>>>> Ideally, we would be able to update without specifying the _rev,
> just
> >>>>>>> posting (or, in this case PUTting) to the document...
> >>>>>>>
> >>>>>>> Thoughts??
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> Luis
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >
>
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
Sorry if my last email was too big :o).
Well, one reason i wanted to avoid doing that is because it didn't
seem as easy to maintain as my original approach but i'll discuss your
suggestion with my team to see what they have to say.
Also, i just couldn't get join to work :o\...
How would you create a view that joins data from those different types
of documents to create a single complete view of a node?
I've read the documentation on view joins but simply could not get it
to work :o\...
Thank you,
Luis
On Tue, Apr 5, 2011 at 9:12 PM, Ryan Ramage <ry...@gmail.com> wrote:
> Luis,
>
> Thats a lot to take in, but a quick suggestion.
>
> Have a parent doc that looks like this:
> {
> id: node1,
> type: node,
> location: blah,
> }
>
> and some 'children' docs that look like this
>
> {
> id: 3232323323223-32323232322-3232,
> timestamp: 1299794532000,
> type: cpu,
> node: node1,
> cpu: 0.94,
> ccores: 4,
> acores: 4,
> cmemory: 4096,
> amemory: 1024
> }
>
> and
> {
> id: 3232323323223-32323232322-3232,
> timestamp: 1299794532000,
> type: disk,
> node: node1,
> disk: 100000
> }
> and
> {
> id: 433432323323223-3232323322332,
> timestamp: 1299794532000,
> type: netio,
> node: node1,
> in: 100,
> out: 200
> }
> and
> {
> id: 323432423432534534-534534-543534534
> timestamp: 1299794532000,
> type: generic,
> node: node1,
> name: "foo",
> value: "bar"
> }
>
> create a status view
> "node_status" : function (doc) {
> if (doc.type != 'node') {
> emit([doc.node, doc.type, doc.timestamp],null);
> }
> }
>
> This allows you to not have to ever update a doc. Just keep inserting.
> Couchdb is good at that.
>
>
>
>
> On Tue, Apr 5, 2011 at 7:20 PM, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>> Thanks for your email Ryan.
>>
>> Let me give you some more information on what i'm trying to do...
>> Essentially, i have to create a "sort of CMDB" system that stores, not only configuration data, but also operational data (so...i guess you could call it a OMDB instead).
>>
>> Either way, my company develops a meta-scheduler that can be used for HPC or Cloud environments. It will guarantee that your resources are used the best way possible, maximizing their usage, based on the policies you set up in it.
>>
>> To do that, our software needs to be aware of how the environment looks and this is why an OMDB piece is very important for us (as it allows us to store information on the environment).
>>
>> Also, our software talks with external resource managers by a protocol we developed more than a dozen years ago called "WIKI" (not as in "wikipedia" but, WIKI as in the hawayan word for fast). That protocol is heavily based around key/value pairs so this is one of the reasons i was EXTREMELY excited to find out that, with CouchDB's "view" functionality, i would be able to map document attributes to more meaningful attributes that our software understands (i.e. map the document's "available_cores" attribute to "ccores" [the "consumable cores" parameter our software understands]).
>>
>> Another important thing to notice is that resources can be off different types: node (for bare metal nodes), vm (for vms running on nodes) and storage (we can actually have more data types but those are enough to exemplify what i'm talking about).
>>
>> This is why i created those "big documents" instead of smaller ones!
>> For instance, each document would represent an entire node (i.e. procs, memory, etc).
>>
>> So my idea was to have an external process initially populate the database with documents representing ALL the nodes we are managing (hence why i started my benchmarks with 100K increments) and OTHER external processes (i.e. other types of resource managers) would update individual attributes in each document.
>>
>> Let's imagine a document with id "node01":
>> These fields would be updated by an agent that collected some of the hardware specs:\
>> ccores: 4 // total cores on machine
>> acores: 4 // available cores on machine
>> cmemory: 4096 // total memory on machine
>> amemory: 1024 // available memory
>> cpuload: 94%
>> This field would be updated by our storage resource manager:
>> GMETRIC["disk"]: 1000000
>> And, for instance, these fields would be updated by a network resource manager:
>> GMETRIC["NETIO"]: { "in":100, "out":200 }
>>
>> So, as you can see, different processes would manage the same document (just different attributes in it).
>>
>> And the REALLY cool thing about the Views is the fact that our customers could VERY easily adapt the database so that it would store THEIR extra data and shove it in a generic parameter that our software woulder understand [i.e. the GMETRIC parameters are generic metrics...).
>>
>> So, based on these requirements, do you have any suggestions on how we should store our data (keeping its structure easy enough for external consumers to maintain it without having to bust their heads figuring out the logic behind the document attributes)?? :o)
>>
>> Thank you!
>> Luis Miguel Silva
>>
>> On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com> wrote:
>>
>>> Luis,
>>>
>>> Having the rev is very important when you update a doc. It lets you
>>> know that your piece of information is out of date. This is a good
>>> thing....
>>>
>>> I am wondering if the way you are modeling your data is not leading
>>> you to do this update with less chance of conflict. See if you can
>>> break your docs into even smaller docs. For example, I noticed from a
>>> prior post you had a lot of Arrays in your docs. If multiple processes
>>> are changing that array, you might be better served by making each
>>> element in the array a separate doc.
>>>
>>> Ryan
>>>
>>> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
>>> <lu...@gmail.com> wrote:
>>>> More or less!
>>>>
>>>> The most common scenario will be:
>>>> - two or more processes writing to the same document, but only to a
>>>> specific attribute (not overwriting the whole document)
>>>>
>>>> If, by any chance, two processes overwrite the same field, i'm ok with
>>>> the last one always winning.
>>>>
>>>> Thanks,
>>>> Luis
>>>>
>>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>>>>> "Ideally, we would be able to update without specifying the _rev, just
>>>>> posting (or, in this case PUTting) to the document..."
>>>>>
>>>>> So you want to blindly overwrite some unknown data?
>>>>>
>>>>> B.
>>>>>
>>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>>>>> Luis,
>>>>>>
>>>>>> Checkout _update handlers:
>>>>>>
>>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Zach
>>>>>>
>>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>>>>> <lu...@gmail.com> wrote:
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>>>>
>>>>>>> Let's image we have to clients that poll a document from the server at
>>>>>>> the same time and get the same _rev.
>>>>>>> Then one of them updates the doc based on the _rev it got:
>>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>>>>> Then another one tries to update the doc based on the same exact _rev:
>>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>>> {"error":"conflict","reason":"Document update conflict."}
>>>>>>> [root@xkitten ~]#
>>>>>>>
>>>>>>> Is there a way to avoid this?! (like...make the update just create a
>>>>>>> new _rev or something)??
>>>>>>>
>>>>>>> Ideally, we would be able to update without specifying the _rev, just
>>>>>>> posting (or, in this case PUTting) to the document...
>>>>>>>
>>>>>>> Thoughts??
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Luis
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: Update conflicts?
Posted by Ryan Ramage <ry...@gmail.com>.
Luis,
Thats a lot to take in, but a quick suggestion.
Have a parent doc that looks like this:
{
id: node1,
type: node,
location: blah,
}
and some 'children' docs that look like this
{
id: 3232323323223-32323232322-3232,
timestamp: 1299794532000,
type: cpu,
node: node1,
cpu: 0.94,
ccores: 4,
acores: 4,
cmemory: 4096,
amemory: 1024
}
and
{
id: 3232323323223-32323232322-3232,
timestamp: 1299794532000,
type: disk,
node: node1,
disk: 100000
}
and
{
id: 433432323323223-3232323322332,
timestamp: 1299794532000,
type: netio,
node: node1,
in: 100,
out: 200
}
and
{
id: 323432423432534534-534534-543534534
timestamp: 1299794532000,
type: generic,
node: node1,
name: "foo",
value: "bar"
}
create a status view
"node_status" : function (doc) {
if (doc.type != 'node') {
emit([doc.node, doc.type, doc.timestamp],null);
}
}
This allows you to not have to ever update a doc. Just keep inserting.
Couchdb is good at that.
On Tue, Apr 5, 2011 at 7:20 PM, Luis Miguel Silva
<lu...@gmail.com> wrote:
> Thanks for your email Ryan.
>
> Let me give you some more information on what i'm trying to do...
> Essentially, i have to create a "sort of CMDB" system that stores, not only configuration data, but also operational data (so...i guess you could call it a OMDB instead).
>
> Either way, my company develops a meta-scheduler that can be used for HPC or Cloud environments. It will guarantee that your resources are used the best way possible, maximizing their usage, based on the policies you set up in it.
>
> To do that, our software needs to be aware of how the environment looks and this is why an OMDB piece is very important for us (as it allows us to store information on the environment).
>
> Also, our software talks with external resource managers by a protocol we developed more than a dozen years ago called "WIKI" (not as in "wikipedia" but, WIKI as in the hawayan word for fast). That protocol is heavily based around key/value pairs so this is one of the reasons i was EXTREMELY excited to find out that, with CouchDB's "view" functionality, i would be able to map document attributes to more meaningful attributes that our software understands (i.e. map the document's "available_cores" attribute to "ccores" [the "consumable cores" parameter our software understands]).
>
> Another important thing to notice is that resources can be off different types: node (for bare metal nodes), vm (for vms running on nodes) and storage (we can actually have more data types but those are enough to exemplify what i'm talking about).
>
> This is why i created those "big documents" instead of smaller ones!
> For instance, each document would represent an entire node (i.e. procs, memory, etc).
>
> So my idea was to have an external process initially populate the database with documents representing ALL the nodes we are managing (hence why i started my benchmarks with 100K increments) and OTHER external processes (i.e. other types of resource managers) would update individual attributes in each document.
>
> Let's imagine a document with id "node01":
> These fields would be updated by an agent that collected some of the hardware specs:\
> ccores: 4 // total cores on machine
> acores: 4 // available cores on machine
> cmemory: 4096 // total memory on machine
> amemory: 1024 // available memory
> cpuload: 94%
> This field would be updated by our storage resource manager:
> GMETRIC["disk"]: 1000000
> And, for instance, these fields would be updated by a network resource manager:
> GMETRIC["NETIO"]: { "in":100, "out":200 }
>
> So, as you can see, different processes would manage the same document (just different attributes in it).
>
> And the REALLY cool thing about the Views is the fact that our customers could VERY easily adapt the database so that it would store THEIR extra data and shove it in a generic parameter that our software woulder understand [i.e. the GMETRIC parameters are generic metrics...).
>
> So, based on these requirements, do you have any suggestions on how we should store our data (keeping its structure easy enough for external consumers to maintain it without having to bust their heads figuring out the logic behind the document attributes)?? :o)
>
> Thank you!
> Luis Miguel Silva
>
> On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com> wrote:
>
>> Luis,
>>
>> Having the rev is very important when you update a doc. It lets you
>> know that your piece of information is out of date. This is a good
>> thing....
>>
>> I am wondering if the way you are modeling your data is not leading
>> you to do this update with less chance of conflict. See if you can
>> break your docs into even smaller docs. For example, I noticed from a
>> prior post you had a lot of Arrays in your docs. If multiple processes
>> are changing that array, you might be better served by making each
>> element in the array a separate doc.
>>
>> Ryan
>>
>> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
>> <lu...@gmail.com> wrote:
>>> More or less!
>>>
>>> The most common scenario will be:
>>> - two or more processes writing to the same document, but only to a
>>> specific attribute (not overwriting the whole document)
>>>
>>> If, by any chance, two processes overwrite the same field, i'm ok with
>>> the last one always winning.
>>>
>>> Thanks,
>>> Luis
>>>
>>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>>>> "Ideally, we would be able to update without specifying the _rev, just
>>>> posting (or, in this case PUTting) to the document..."
>>>>
>>>> So you want to blindly overwrite some unknown data?
>>>>
>>>> B.
>>>>
>>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>>>> Luis,
>>>>>
>>>>> Checkout _update handlers:
>>>>>
>>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Zach
>>>>>
>>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>>>> <lu...@gmail.com> wrote:
>>>>>> Dear all,
>>>>>>
>>>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>>>
>>>>>> Let's image we have to clients that poll a document from the server at
>>>>>> the same time and get the same _rev.
>>>>>> Then one of them updates the doc based on the _rev it got:
>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>>>> Then another one tries to update the doc based on the same exact _rev:
>>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>>> {"error":"conflict","reason":"Document update conflict."}
>>>>>> [root@xkitten ~]#
>>>>>>
>>>>>> Is there a way to avoid this?! (like...make the update just create a
>>>>>> new _rev or something)??
>>>>>>
>>>>>> Ideally, we would be able to update without specifying the _rev, just
>>>>>> posting (or, in this case PUTting) to the document...
>>>>>>
>>>>>> Thoughts??
>>>>>>
>>>>>> Thank you,
>>>>>> Luis
>>>>>>
>>>>>
>>>>
>>>
>
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
Thanks for your email Ryan.
Let me give you some more information on what i'm trying to do...
Essentially, i have to create a "sort of CMDB" system that stores, not only configuration data, but also operational data (so...i guess you could call it a OMDB instead).
Either way, my company develops a meta-scheduler that can be used for HPC or Cloud environments. It will guarantee that your resources are used the best way possible, maximizing their usage, based on the policies you set up in it.
To do that, our software needs to be aware of how the environment looks and this is why an OMDB piece is very important for us (as it allows us to store information on the environment).
Also, our software talks with external resource managers by a protocol we developed more than a dozen years ago called "WIKI" (not as in "wikipedia" but, WIKI as in the hawayan word for fast). That protocol is heavily based around key/value pairs so this is one of the reasons i was EXTREMELY excited to find out that, with CouchDB's "view" functionality, i would be able to map document attributes to more meaningful attributes that our software understands (i.e. map the document's "available_cores" attribute to "ccores" [the "consumable cores" parameter our software understands]).
Another important thing to notice is that resources can be off different types: node (for bare metal nodes), vm (for vms running on nodes) and storage (we can actually have more data types but those are enough to exemplify what i'm talking about).
This is why i created those "big documents" instead of smaller ones!
For instance, each document would represent an entire node (i.e. procs, memory, etc).
So my idea was to have an external process initially populate the database with documents representing ALL the nodes we are managing (hence why i started my benchmarks with 100K increments) and OTHER external processes (i.e. other types of resource managers) would update individual attributes in each document.
Let's imagine a document with id "node01":
These fields would be updated by an agent that collected some of the hardware specs:
ccores: 4 // total cores on machine
acores: 4 // available cores on machine
cmemory: 4096 // total memory on machine
amemory: 1024 // available memory
cpuload: 94%
This field would be updated by our storage resource manager:
GMETRIC["disk"]: 1000000
And, for instance, these fields would be updated by a network resource manager:
GMETRIC["NETIO"]: { "in":100, "out":200 }
So, as you can see, different processes would manage the same document (just different attributes in it).
And the REALLY cool thing about the Views is the fact that our customers could VERY easily adapt the database so that it would store THEIR extra data and shove it in a generic parameter that our software woulder understand [i.e. the GMETRIC parameters are generic metrics...).
So, based on these requirements, do you have any suggestions on how we should store our data (keeping its structure easy enough for external consumers to maintain it without having to bust their heads figuring out the logic behind the document attributes)?? :o)
Thank you!
Luis Miguel Silva
On Apr 5, 2011, at 6:45 PM, Ryan Ramage <ry...@gmail.com> wrote:
> Luis,
>
> Having the rev is very important when you update a doc. It lets you
> know that your piece of information is out of date. This is a good
> thing....
>
> I am wondering if the way you are modeling your data is not leading
> you to do this update with less chance of conflict. See if you can
> break your docs into even smaller docs. For example, I noticed from a
> prior post you had a lot of Arrays in your docs. If multiple processes
> are changing that array, you might be better served by making each
> element in the array a separate doc.
>
> Ryan
>
> On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>> More or less!
>>
>> The most common scenario will be:
>> - two or more processes writing to the same document, but only to a
>> specific attribute (not overwriting the whole document)
>>
>> If, by any chance, two processes overwrite the same field, i'm ok with
>> the last one always winning.
>>
>> Thanks,
>> Luis
>>
>> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>>> "Ideally, we would be able to update without specifying the _rev, just
>>> posting (or, in this case PUTting) to the document..."
>>>
>>> So you want to blindly overwrite some unknown data?
>>>
>>> B.
>>>
>>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>>> Luis,
>>>>
>>>> Checkout _update handlers:
>>>>
>>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Zach
>>>>
>>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>>> <lu...@gmail.com> wrote:
>>>>> Dear all,
>>>>>
>>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>>
>>>>> Let's image we have to clients that poll a document from the server at
>>>>> the same time and get the same _rev.
>>>>> Then one of them updates the doc based on the _rev it got:
>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>>> Then another one tries to update the doc based on the same exact _rev:
>>>>> [root@xkitten ~]# curl -X PUT -d
>>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>>> {"error":"conflict","reason":"Document update conflict."}
>>>>> [root@xkitten ~]#
>>>>>
>>>>> Is there a way to avoid this?! (like...make the update just create a
>>>>> new _rev or something)??
>>>>>
>>>>> Ideally, we would be able to update without specifying the _rev, just
>>>>> posting (or, in this case PUTting) to the document...
>>>>>
>>>>> Thoughts??
>>>>>
>>>>> Thank you,
>>>>> Luis
>>>>>
>>>>
>>>
>>
Re: Update conflicts?
Posted by Ryan Ramage <ry...@gmail.com>.
Luis,
Having the rev is very important when you update a doc. It lets you
know that your piece of information is out of date. This is a good
thing....
I am wondering if the way you are modeling your data is not leading
you to do this update with less chance of conflict. See if you can
break your docs into even smaller docs. For example, I noticed from a
prior post you had a lot of Arrays in your docs. If multiple processes
are changing that array, you might be better served by making each
element in the array a separate doc.
Ryan
On Tue, Apr 5, 2011 at 4:41 PM, Luis Miguel Silva
<lu...@gmail.com> wrote:
> More or less!
>
> The most common scenario will be:
> - two or more processes writing to the same document, but only to a
> specific attribute (not overwriting the whole document)
>
> If, by any chance, two processes overwrite the same field, i'm ok with
> the last one always winning.
>
> Thanks,
> Luis
>
> On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
>> "Ideally, we would be able to update without specifying the _rev, just
>> posting (or, in this case PUTting) to the document..."
>>
>> So you want to blindly overwrite some unknown data?
>>
>> B.
>>
>> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>>> Luis,
>>>
>>> Checkout _update handlers:
>>>
>>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>>
>>>
>>> Cheers,
>>>
>>> Zach
>>>
>>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>>> <lu...@gmail.com> wrote:
>>>> Dear all,
>>>>
>>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>>
>>>> Let's image we have to clients that poll a document from the server at
>>>> the same time and get the same _rev.
>>>> Then one of them updates the doc based on the _rev it got:
>>>> [root@xkitten ~]# curl -X PUT -d
>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>>> Then another one tries to update the doc based on the same exact _rev:
>>>> [root@xkitten ~]# curl -X PUT -d
>>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>>> http://localhost:5984/benchmark/test?conflicts=true
>>>> {"error":"conflict","reason":"Document update conflict."}
>>>> [root@xkitten ~]#
>>>>
>>>> Is there a way to avoid this?! (like...make the update just create a
>>>> new _rev or something)??
>>>>
>>>> Ideally, we would be able to update without specifying the _rev, just
>>>> posting (or, in this case PUTting) to the document...
>>>>
>>>> Thoughts??
>>>>
>>>> Thank you,
>>>> Luis
>>>>
>>>
>>
>
Re: Update conflicts?
Posted by Luis Miguel Silva <lu...@gmail.com>.
More or less!
The most common scenario will be:
- two or more processes writing to the same document, but only to a
specific attribute (not overwriting the whole document)
If, by any chance, two processes overwrite the same field, i'm ok with
the last one always winning.
Thanks,
Luis
On Tue, Apr 5, 2011 at 4:26 PM, Robert Newson <ro...@gmail.com> wrote:
> "Ideally, we would be able to update without specifying the _rev, just
> posting (or, in this case PUTting) to the document..."
>
> So you want to blindly overwrite some unknown data?
>
> B.
>
> On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
>> Luis,
>>
>> Checkout _update handlers:
>>
>> http://wiki.apache.org/couchdb/Document_Update_Handlers
>>
>>
>> Cheers,
>>
>> Zach
>>
>> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
>> <lu...@gmail.com> wrote:
>>> Dear all,
>>>
>>> I'm trying to play around with updates and i'm bumping into some problems.
>>>
>>> Let's image we have to clients that poll a document from the server at
>>> the same time and get the same _rev.
>>> Then one of them updates the doc based on the _rev it got:
>>> [root@xkitten ~]# curl -X PUT -d
>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>>> http://localhost:5984/benchmark/test?conflicts=true
>>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>>> Then another one tries to update the doc based on the same exact _rev:
>>> [root@xkitten ~]# curl -X PUT -d
>>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>>> http://localhost:5984/benchmark/test?conflicts=true
>>> {"error":"conflict","reason":"Document update conflict."}
>>> [root@xkitten ~]#
>>>
>>> Is there a way to avoid this?! (like...make the update just create a
>>> new _rev or something)??
>>>
>>> Ideally, we would be able to update without specifying the _rev, just
>>> posting (or, in this case PUTting) to the document...
>>>
>>> Thoughts??
>>>
>>> Thank you,
>>> Luis
>>>
>>
>
Re: Update conflicts?
Posted by Robert Newson <ro...@gmail.com>.
"Ideally, we would be able to update without specifying the _rev, just
posting (or, in this case PUTting) to the document..."
So you want to blindly overwrite some unknown data?
B.
On 5 April 2011 22:57, Zachary Zolton <za...@gmail.com> wrote:
> Luis,
>
> Checkout _update handlers:
>
> http://wiki.apache.org/couchdb/Document_Update_Handlers
>
>
> Cheers,
>
> Zach
>
> On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
> <lu...@gmail.com> wrote:
>> Dear all,
>>
>> I'm trying to play around with updates and i'm bumping into some problems.
>>
>> Let's image we have to clients that poll a document from the server at
>> the same time and get the same _rev.
>> Then one of them updates the doc based on the _rev it got:
>> [root@xkitten ~]# curl -X PUT -d
>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
>> http://localhost:5984/benchmark/test?conflicts=true
>> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
>> Then another one tries to update the doc based on the same exact _rev:
>> [root@xkitten ~]# curl -X PUT -d
>> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
>> http://localhost:5984/benchmark/test?conflicts=true
>> {"error":"conflict","reason":"Document update conflict."}
>> [root@xkitten ~]#
>>
>> Is there a way to avoid this?! (like...make the update just create a
>> new _rev or something)??
>>
>> Ideally, we would be able to update without specifying the _rev, just
>> posting (or, in this case PUTting) to the document...
>>
>> Thoughts??
>>
>> Thank you,
>> Luis
>>
>
Re: Update conflicts?
Posted by Zachary Zolton <za...@gmail.com>.
Luis,
Checkout _update handlers:
http://wiki.apache.org/couchdb/Document_Update_Handlers
Cheers,
Zach
On Tue, Apr 5, 2011 at 4:46 PM, Luis Miguel Silva
<lu...@gmail.com> wrote:
> Dear all,
>
> I'm trying to play around with updates and i'm bumping into some problems.
>
> Let's image we have to clients that poll a document from the server at
> the same time and get the same _rev.
> Then one of them updates the doc based on the _rev it got:
> [root@xkitten ~]# curl -X PUT -d
> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello2":"fred2"}'
> http://localhost:5984/benchmark/test?conflicts=true
> {"ok":true,"id":"test","rev":"4-03640ebafbb4fcaf127844671f8e2de7"}
> Then another one tries to update the doc based on the same exact _rev:
> [root@xkitten ~]# curl -X PUT -d
> '{"_rev":"3-0d519bcf08130bf784f3c35d79760740","hello3":"fred3"}'
> http://localhost:5984/benchmark/test?conflicts=true
> {"error":"conflict","reason":"Document update conflict."}
> [root@xkitten ~]#
>
> Is there a way to avoid this?! (like...make the update just create a
> new _rev or something)??
>
> Ideally, we would be able to update without specifying the _rev, just
> posting (or, in this case PUTting) to the document...
>
> Thoughts??
>
> Thank you,
> Luis
>