You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by "Szabo, Viktor" <Vi...@morganstanley.com> on 2012/02/20 22:16:12 UTC
RE: possible compact bug in 1.1.1
Hi,
An application we have is using bulk writes to insert objects into the database, and occasionally some
documents get lost. It turns out this happens for the very same reason described below. If we're
re-insering a doc using an id that has been deleted earlier, if the contents match the payload that was
saved when the id got used for the first time, the doc remains in "deleted" state and the client
gets a misleading response. This only happens when the database is compacted after the delete operation
is executed.
I still feel that this is pretty nasty - what do you guys think?
Looking at the reply coming from _bulk_docs, there's no way I can tell if I can read back my doc from the database or not.
Below are the steps to reproduce.
Thanks,
Viktor
$ curl -X POST -H "Content-type: application/json" http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key" : "value" } ] }'
[{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
$ curl -X GET http://server/database/bug
{"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"}
$ curl -X DELETE http://server/database/bug?rev=1-59414e77c768bc202142ac82c2f129de
{"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
$ curl -X GET http://server/database/bug
{"error":"not_found","reason":"deleted"}
$ curl -X POST -H "Content-type: application/json" http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key" : "value" } ] }'
[{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
$ curl -X GET http://server/database/bug
{"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"}
$ curl -X DELETE http://server/database/bug?rev=3-82b9390af5b7c003e03c0dd7e6aac45a
{"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
$ curl -X POST -H "Content-type: application/json" http://server/database/_compact
{"ok":true}
$ curl -X GET http://server/database/bug
{"error":"not_found","reason":"deleted"}
$ curl -X POST -H "Content-type: application/json" http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key" : "value" } ] }'
[{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
$ curl -X GET http://server/database/bug
{"error":"not_found","reason":"deleted"}
-----Original Message-----
From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
Sent: 23 November 2011 14:38
To: user@couchdb.apache.org
Subject: Re: possible compact bug in 1.1.1
When a new document is created, the response is 201, not 200.
If the posted document is identical to the one saved, the revision is the same (otherwise it would have been a conflict). So the revision you have is not obsolete.
Marcello
2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
<Vi...@morganstanley.com>:
> I'd rather know about the fact that I haven't just successfully
> created a doc, but re-submitted a revision that was already known -
> and is already obsolete as revisions with higher version numbers already exist.
>
> Cheers,
> Viktor
>
> -----Original Message-----
> From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> Sent: 23 November 2011 13:41
> To: user@couchdb.apache.org
> Subject: Re: possible compact bug in 1.1.1
>
> Can you elaborate on why you want a conflict?
> I find it confusing to have a conflict when, in fact, there can't be any conflict since nothing has changed.
>
> Marcello
>
> 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> <Vi...@morganstanley.com>:
>> Thanks Paul, this makes sense.
>>
>> If it counts, I vote for forcing a conflict ;)
>>
>> Cheers,
>> Viktor
>>
>> -----Original Message-----
>> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
>> Sent: 22 November 2011 20:54
>> To: user@couchdb.apache.org
>> Subject: Re: possible compact bug in 1.1.1
>>
>>
>> Your example here is actually hitting a very specific edge case as demonstrated by Marcello's test. As of many versions ago, revisions are generated using a hashing scheme of the document contents. In your particular case the requests you're issuing contain the same identical data in such a way that CouchDB will generate a revision of the doc.
>>
>> Given this, we then have to look at how this plays into replication.
>> Basically, when we merge the revision trees we get to the case where it's "oh, we already have this version, cool" because we do already have this version.
>>
>> Whether or not that behavior is best, or if we should force a conflict if we don't add a leaf during a write is another question. In other words, the system is working fine, but this particular behavior can be a bit unexpected.
>>
>> ---------------------------------------------------------------------
>> -
>> ----
>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
>
> ----------------------------------------------------------------------
> ----
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
RE: possible compact bug in 1.1.1
Posted by "Szabo, Viktor" <Vi...@morganstanley.com>.
Hi CGS,
Thanks for looking into this.
There's nothing in the log:
[Tue, 21 Feb 2012 09:35:44 GMT] [debug] [<0.10552.48>] 'POST' /database/_bulk_docs {1,1} from "114.24.42.22"
[Tue, 21 Feb 2012 09:35:44 GMT] [info] [<0.10552.48>] 114.24.42.22 - - 'POST' /database/_bulk_docs 201
Thanks,
Viktor
-----Original Message-----
From: CGS [mailto:cgsmcmlxxv@gmail.com]
Sent: 21 February 2012 00:33
To: user@couchdb.apache.org
Subject: Re: possible compact bug in 1.1.1
I would dare another question, if I may. Could you tell me if there is anything in the CouchDB log which would give a hint that the insertion wasn't successful? Thank you.
CGS
On Tue, Feb 21, 2012 at 12:17 AM, CGS <cg...@gmail.com> wrote:
> It does matter because it may be that during compaction, the bulk
> insertion may see no document with that name, so, it reports back the
> first revision for the document (1-...). As far as I know (and I might
> be wrong here), the compaction keeps the tombstone for the deleted
> document and that is a revision for the document. That means, your
> document should have a revision higher than 1-... after the
> re-insertion because the document exists even if marked as deleted.
> But if you say that happens after the compaction was completed, then
> it is clear the scenario is not the same with what I presented here.
>
> In your final insertion, you got the exact first revision as you would
> create it anew. I usually take that as an indicator that something
> went wrong during the insertion. I am not saying it's not a bug.
>
> I am not a developer (just helping occasionally in maintaining the
> documentation), but I try to understand the bugs for the next user to
> be aware of such and how to catch it if the bug is not eliminated. So,
> my question is purely for information and understanding. The rest
> depends on the developers.
>
> Thank you for your understanding and your patience in answering my
> question. I will follow this discussion and its outcome.
>
> CGS
>
>
>
>
>
> On Mon, Feb 20, 2012 at 11:42 PM, Szabo, Viktor <
> Viktor.Szabo@morganstanley.com> wrote:
>
>> Yes, the re-insertion took place after the compaction completed.
>>
>> Should it matter? Does it matter?
>>
>> Thanks,
>> Viktor
>>
>> -----Original Message-----
>> From: CGS [mailto:cgsmcmlxxv@gmail.com]
>> Sent: 20 February 2012 23:22
>> To: user@couchdb.apache.org
>> Subject: Re: possible compact bug in 1.1.1
>>
>> Hi,
>>
>> Was the compaction finished when you tried to insert the document?
>>
>> CGS
>>
>>
>>
>>
>> On Mon, Feb 20, 2012 at 10:16 PM, Szabo, Viktor <
>> Viktor.Szabo@morganstanley.com> wrote:
>>
>> > Hi,
>> >
>> > An application we have is using bulk writes to insert objects into
>> > the database, and occasionally some documents get lost. It turns
>> > out this happens for the very same reason described below. If we're
>> > re-insering a doc using an id that has been deleted earlier, if the
>> > contents match the payload that was saved when the id got used for
>> > the first time, the doc remains in "deleted" state and the client
>> > gets a misleading response. This only happens when the database is
>> > compacted after the delete operation is executed.
>> >
>> > I still feel that this is pretty nasty - what do you guys think?
>> >
>> > Looking at the reply coming from _bulk_docs, there's no way I can
>> > tell if I can read back my doc from the database or not.
>> >
>> > Below are the steps to reproduce.
>> >
>> > Thanks,
>> > Viktor
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug",
>> "key"
>> > : "value" } ] }'
>> > [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"
>> > }
>> >
>> > $ curl -X DELETE
>> > http://server/database/bug?rev=1-59414e77c768bc202142ac82c2f129de
>> > {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"error":"not_found","reason":"deleted"}
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug",
>> "key"
>> > : "value" } ] }'
>> > [{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"
>> > }
>> >
>> > $ curl -X DELETE
>> > http://server/database/bug?rev=3-82b9390af5b7c003e03c0dd7e6aac45a
>> > {"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_compact
>> > {"ok":true}
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"error":"not_found","reason":"deleted"}
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug",
>> "key"
>> > : "value" } ] }'
>> > [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"error":"not_found","reason":"deleted"}
>> >
>> >
>> > -----Original Message-----
>> > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
>> > Sent: 23 November 2011 14:38
>> > To: user@couchdb.apache.org
>> > Subject: Re: possible compact bug in 1.1.1
>> >
>> > When a new document is created, the response is 201, not 200.
>> > If the posted document is identical to the one saved, the revision
>> > is the same (otherwise it would have been a conflict). So the
>> > revision you have is not obsolete.
>> >
>> > Marcello
>> >
>> > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
>> > <Vi...@morganstanley.com>:
>> > > I'd rather know about the fact that I haven't just successfully
>> > > created a doc, but re-submitted a revision that was already known
>> > > - and is already obsolete as revisions with higher version
>> > > numbers already
>> > exist.
>> > >
>> > > Cheers,
>> > > Viktor
>> > >
>> > > -----Original Message-----
>> > > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
>> > > Sent: 23 November 2011 13:41
>> > > To: user@couchdb.apache.org
>> > > Subject: Re: possible compact bug in 1.1.1
>> > >
>> > > Can you elaborate on why you want a conflict?
>> > > I find it confusing to have a conflict when, in fact, there can't
>> > > be any
>> > conflict since nothing has changed.
>> > >
>> > > Marcello
>> > >
>> > > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
>> > > <Vi...@morganstanley.com>:
>> > >> Thanks Paul, this makes sense.
>> > >>
>> > >> If it counts, I vote for forcing a conflict ;)
>> > >>
>> > >> Cheers,
>> > >> Viktor
>> > >>
>> > >> -----Original Message-----
>> > >> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
>> > >> Sent: 22 November 2011 20:54
>> > >> To: user@couchdb.apache.org
>> > >> Subject: Re: possible compact bug in 1.1.1
>> > >>
>> > >>
>> > >> Your example here is actually hitting a very specific edge case
>> > >> as
>> > demonstrated by Marcello's test. As of many versions ago, revisions
>> > are generated using a hashing scheme of the document contents. In
>> > your particular case the requests you're issuing contain the same
>> > identical data in such a way that CouchDB will generate a revision of the doc.
>> > >>
>> > >> Given this, we then have to look at how this plays into replication.
>> > >> Basically, when we merge the revision trees we get to the case
>> > >> where
>> > it's "oh, we already have this version, cool" because we do already
>> > have this version.
>> > >>
>> > >> Whether or not that behavior is best, or if we should force a
>> > >> conflict
>> > if we don't add a leaf during a write is another question. In other
>> > words, the system is working fine, but this particular behavior can
>> > be a bit unexpected.
>> > >>
>> > >> ----------------------------------------------------------------
>> > >> ---
>> > >> --
>> > >> -
>> > >> ----
>> > >> NOTICE: Morgan Stanley is not acting as a municipal advisor and
>> > >> the
>> > opinions or views contained herein are not intended to be, and do
>> > not constitute, advice within the meaning of Section 975 of the
>> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
>> > have received this communication in error, please destroy all
>> > electronic and paper copies and notify the sender immediately.
>> > Mistransmission is not intended to waive confidentiality or
>> > privilege. Morgan Stanley reserves the right, to the extent
>> > permitted under applicable law, to monitor electronic
>> > communications. This message is subject to terms available at the
>> > following
>> > link: http://www.morganstanley.com/disclaimers. If you cannot
>> > access these links, please notify us by reply message and we will
>> > send the contents to you. By messaging with Morgan Stanley you
>> > consent to the foregoing.
>> > >
>> > > -----------------------------------------------------------------
>> > > ---
>> > > --
>> > > ----
>> > > NOTICE: Morgan Stanley is not acting as a municipal advisor and
>> > > the
>> > opinions or views contained herein are not intended to be, and do
>> > not constitute, advice within the meaning of Section 975 of the
>> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
>> > have received this communication in error, please destroy all
>> > electronic and paper copies and notify the sender immediately.
>> > Mistransmission is not intended to waive confidentiality or
>> > privilege. Morgan Stanley reserves the right, to the extent
>> > permitted under applicable law, to monitor electronic
>> > communications. This message is subject to terms available at the
>> > following
>> > link: http://www.morganstanley.com/disclaimers. If you cannot
>> > access these links, please notify us by reply message and we will
>> > send the contents to you. By messaging with Morgan Stanley you
>> > consent to the foregoing.
>> >
>> > -------------------------------------------------------------------
>> > ---
>> > ----
>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> > opinions or views contained herein are not intended to be, and do
>> > not constitute, advice within the meaning of Section 975 of the
>> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
>> > have received this communication in error, please destroy all
>> > electronic and paper copies and notify the sender immediately.
>> > Mistransmission is not intended to waive confidentiality or
>> > privilege. Morgan Stanley reserves the right, to the extent
>> > permitted under applicable law, to monitor electronic
>> > communications. This message is subject to terms available at the
>> > following
>> > link: http://www.morganstanley.com/disclaimers. If you cannot
>> > access these links, please notify us by reply message and we will
>> > send the contents to you. By messaging with Morgan Stanley you
>> > consent to the foregoing.
>> >
>>
>> ---------------------------------------------------------------------
>> -----
>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> opinions or views contained herein are not intended to be, and do not
>> constitute, advice within the meaning of Section 975 of the
>> Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
>> have received this communication in error, please destroy all
>> electronic and paper copies and notify the sender immediately.
>> Mistransmission is not intended to waive confidentiality or
>> privilege. Morgan Stanley reserves the right, to the extent permitted
>> under applicable law, to monitor electronic communications. This
>> message is subject to terms available at the following
>> link: http://www.morganstanley.com/disclaimers. If you cannot access
>> these links, please notify us by reply message and we will send the
>> contents to you. By messaging with Morgan Stanley you consent to the
>> foregoing.
>>
>
>
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: possible compact bug in 1.1.1
Posted by CGS <cg...@gmail.com>.
I would dare another question, if I may. Could you tell me if there is
anything in the CouchDB log which would give a hint that the insertion
wasn't successful? Thank you.
CGS
On Tue, Feb 21, 2012 at 12:17 AM, CGS <cg...@gmail.com> wrote:
> It does matter because it may be that during compaction, the bulk
> insertion may see no document with that name, so, it reports back the first
> revision for the document (1-...). As far as I know (and I might be wrong
> here), the compaction keeps the tombstone for the deleted document and that
> is a revision for the document. That means, your document should have a
> revision higher than 1-... after the re-insertion because the document
> exists even if marked as deleted. But if you say that happens after the
> compaction was completed, then it is clear the scenario is not the same
> with what I presented here.
>
> In your final insertion, you got the exact first revision as you would
> create it anew. I usually take that as an indicator that something went
> wrong during the insertion. I am not saying it's not a bug.
>
> I am not a developer (just helping occasionally in maintaining the
> documentation), but I try to understand the bugs for the next user to be
> aware of such and how to catch it if the bug is not eliminated. So, my
> question is purely for information and understanding. The rest depends on
> the developers.
>
> Thank you for your understanding and your patience in answering my
> question. I will follow this discussion and its outcome.
>
> CGS
>
>
>
>
>
> On Mon, Feb 20, 2012 at 11:42 PM, Szabo, Viktor <
> Viktor.Szabo@morganstanley.com> wrote:
>
>> Yes, the re-insertion took place after the compaction completed.
>>
>> Should it matter? Does it matter?
>>
>> Thanks,
>> Viktor
>>
>> -----Original Message-----
>> From: CGS [mailto:cgsmcmlxxv@gmail.com]
>> Sent: 20 February 2012 23:22
>> To: user@couchdb.apache.org
>> Subject: Re: possible compact bug in 1.1.1
>>
>> Hi,
>>
>> Was the compaction finished when you tried to insert the document?
>>
>> CGS
>>
>>
>>
>>
>> On Mon, Feb 20, 2012 at 10:16 PM, Szabo, Viktor <
>> Viktor.Szabo@morganstanley.com> wrote:
>>
>> > Hi,
>> >
>> > An application we have is using bulk writes to insert objects into the
>> > database, and occasionally some documents get lost. It turns out this
>> > happens for the very same reason described below. If we're re-insering
>> > a doc using an id that has been deleted earlier, if the contents match
>> > the payload that was saved when the id got used for the first time,
>> > the doc remains in "deleted" state and the client gets a misleading
>> > response. This only happens when the database is compacted after the
>> > delete operation is executed.
>> >
>> > I still feel that this is pretty nasty - what do you guys think?
>> >
>> > Looking at the reply coming from _bulk_docs, there's no way I can tell
>> > if I can read back my doc from the database or not.
>> >
>> > Below are the steps to reproduce.
>> >
>> > Thanks,
>> > Viktor
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug",
>> "key"
>> > : "value" } ] }'
>> > [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"
>> > }
>> >
>> > $ curl -X DELETE
>> > http://server/database/bug?rev=1-59414e77c768bc202142ac82c2f129de
>> > {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"error":"not_found","reason":"deleted"}
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug",
>> "key"
>> > : "value" } ] }'
>> > [{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"
>> > }
>> >
>> > $ curl -X DELETE
>> > http://server/database/bug?rev=3-82b9390af5b7c003e03c0dd7e6aac45a
>> > {"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_compact
>> > {"ok":true}
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"error":"not_found","reason":"deleted"}
>> >
>> > $ curl -X POST -H "Content-type: application/json"
>> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug",
>> "key"
>> > : "value" } ] }'
>> > [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>> >
>> > $ curl -X GET http://server/database/bug
>> > {"error":"not_found","reason":"deleted"}
>> >
>> >
>> > -----Original Message-----
>> > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
>> > Sent: 23 November 2011 14:38
>> > To: user@couchdb.apache.org
>> > Subject: Re: possible compact bug in 1.1.1
>> >
>> > When a new document is created, the response is 201, not 200.
>> > If the posted document is identical to the one saved, the revision is
>> > the same (otherwise it would have been a conflict). So the revision
>> > you have is not obsolete.
>> >
>> > Marcello
>> >
>> > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
>> > <Vi...@morganstanley.com>:
>> > > I'd rather know about the fact that I haven't just successfully
>> > > created a doc, but re-submitted a revision that was already known -
>> > > and is already obsolete as revisions with higher version numbers
>> > > already
>> > exist.
>> > >
>> > > Cheers,
>> > > Viktor
>> > >
>> > > -----Original Message-----
>> > > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
>> > > Sent: 23 November 2011 13:41
>> > > To: user@couchdb.apache.org
>> > > Subject: Re: possible compact bug in 1.1.1
>> > >
>> > > Can you elaborate on why you want a conflict?
>> > > I find it confusing to have a conflict when, in fact, there can't be
>> > > any
>> > conflict since nothing has changed.
>> > >
>> > > Marcello
>> > >
>> > > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
>> > > <Vi...@morganstanley.com>:
>> > >> Thanks Paul, this makes sense.
>> > >>
>> > >> If it counts, I vote for forcing a conflict ;)
>> > >>
>> > >> Cheers,
>> > >> Viktor
>> > >>
>> > >> -----Original Message-----
>> > >> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
>> > >> Sent: 22 November 2011 20:54
>> > >> To: user@couchdb.apache.org
>> > >> Subject: Re: possible compact bug in 1.1.1
>> > >>
>> > >>
>> > >> Your example here is actually hitting a very specific edge case as
>> > demonstrated by Marcello's test. As of many versions ago, revisions
>> > are generated using a hashing scheme of the document contents. In your
>> > particular case the requests you're issuing contain the same identical
>> > data in such a way that CouchDB will generate a revision of the doc.
>> > >>
>> > >> Given this, we then have to look at how this plays into replication.
>> > >> Basically, when we merge the revision trees we get to the case
>> > >> where
>> > it's "oh, we already have this version, cool" because we do already
>> > have this version.
>> > >>
>> > >> Whether or not that behavior is best, or if we should force a
>> > >> conflict
>> > if we don't add a leaf during a write is another question. In other
>> > words, the system is working fine, but this particular behavior can be
>> > a bit unexpected.
>> > >>
>> > >> -------------------------------------------------------------------
>> > >> --
>> > >> -
>> > >> ----
>> > >> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> > opinions or views contained herein are not intended to be, and do not
>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>> > Wall Street Reform and Consumer Protection Act. If you have received
>> > this communication in error, please destroy all electronic and paper
>> > copies and notify the sender immediately. Mistransmission is not
>> > intended to waive confidentiality or privilege. Morgan Stanley
>> > reserves the right, to the extent permitted under applicable law, to
>> > monitor electronic communications. This message is subject to terms
>> > available at the following
>> > link: http://www.morganstanley.com/disclaimers. If you cannot access
>> > these links, please notify us by reply message and we will send the
>> > contents to you. By messaging with Morgan Stanley you consent to the
>> > foregoing.
>> > >
>> > > --------------------------------------------------------------------
>> > > --
>> > > ----
>> > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> > opinions or views contained herein are not intended to be, and do not
>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>> > Wall Street Reform and Consumer Protection Act. If you have received
>> > this communication in error, please destroy all electronic and paper
>> > copies and notify the sender immediately. Mistransmission is not
>> > intended to waive confidentiality or privilege. Morgan Stanley
>> > reserves the right, to the extent permitted under applicable law, to
>> > monitor electronic communications. This message is subject to terms
>> > available at the following
>> > link: http://www.morganstanley.com/disclaimers. If you cannot access
>> > these links, please notify us by reply message and we will send the
>> > contents to you. By messaging with Morgan Stanley you consent to the
>> > foregoing.
>> >
>> > ----------------------------------------------------------------------
>> > ----
>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> > opinions or views contained herein are not intended to be, and do not
>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>> > Wall Street Reform and Consumer Protection Act. If you have received
>> > this communication in error, please destroy all electronic and paper
>> > copies and notify the sender immediately. Mistransmission is not
>> > intended to waive confidentiality or privilege. Morgan Stanley
>> > reserves the right, to the extent permitted under applicable law, to
>> > monitor electronic communications. This message is subject to terms
>> > available at the following
>> > link: http://www.morganstanley.com/disclaimers. If you cannot access
>> > these links, please notify us by reply message and we will send the
>> > contents to you. By messaging with Morgan Stanley you consent to the
>> > foregoing.
>> >
>>
>> --------------------------------------------------------------------------
>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> opinions or views contained herein are not intended to be, and do not
>> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
>> Street Reform and Consumer Protection Act. If you have received this
>> communication in error, please destroy all electronic and paper copies and
>> notify the sender immediately. Mistransmission is not intended to waive
>> confidentiality or privilege. Morgan Stanley reserves the right, to the
>> extent permitted under applicable law, to monitor electronic
>> communications. This message is subject to terms available at the following
>> link: http://www.morganstanley.com/disclaimers. If you cannot access
>> these links, please notify us by reply message and we will send the
>> contents to you. By messaging with Morgan Stanley you consent to the
>> foregoing.
>>
>
>
Re: possible compact bug in 1.1.1
Posted by CGS <cg...@gmail.com>.
It does matter because it may be that during compaction, the bulk insertion
may see no document with that name, so, it reports back the first revision
for the document (1-...). As far as I know (and I might be wrong here), the
compaction keeps the tombstone for the deleted document and that is a
revision for the document. That means, your document should have a revision
higher than 1-... after the re-insertion because the document exists even
if marked as deleted. But if you say that happens after the compaction was
completed, then it is clear the scenario is not the same with what I
presented here.
In your final insertion, you got the exact first revision as you would
create it anew. I usually take that as an indicator that something went
wrong during the insertion. I am not saying it's not a bug.
I am not a developer (just helping occasionally in maintaining the
documentation), but I try to understand the bugs for the next user to be
aware of such and how to catch it if the bug is not eliminated. So, my
question is purely for information and understanding. The rest depends on
the developers.
Thank you for your understanding and your patience in answering my
question. I will follow this discussion and its outcome.
CGS
On Mon, Feb 20, 2012 at 11:42 PM, Szabo, Viktor <
Viktor.Szabo@morganstanley.com> wrote:
> Yes, the re-insertion took place after the compaction completed.
>
> Should it matter? Does it matter?
>
> Thanks,
> Viktor
>
> -----Original Message-----
> From: CGS [mailto:cgsmcmlxxv@gmail.com]
> Sent: 20 February 2012 23:22
> To: user@couchdb.apache.org
> Subject: Re: possible compact bug in 1.1.1
>
> Hi,
>
> Was the compaction finished when you tried to insert the document?
>
> CGS
>
>
>
>
> On Mon, Feb 20, 2012 at 10:16 PM, Szabo, Viktor <
> Viktor.Szabo@morganstanley.com> wrote:
>
> > Hi,
> >
> > An application we have is using bulk writes to insert objects into the
> > database, and occasionally some documents get lost. It turns out this
> > happens for the very same reason described below. If we're re-insering
> > a doc using an id that has been deleted earlier, if the contents match
> > the payload that was saved when the id got used for the first time,
> > the doc remains in "deleted" state and the client gets a misleading
> > response. This only happens when the database is compacted after the
> > delete operation is executed.
> >
> > I still feel that this is pretty nasty - what do you guys think?
> >
> > Looking at the reply coming from _bulk_docs, there's no way I can tell
> > if I can read back my doc from the database or not.
> >
> > Below are the steps to reproduce.
> >
> > Thanks,
> > Viktor
> >
> > $ curl -X POST -H "Content-type: application/json"
> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> > : "value" } ] }'
> > [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
> >
> > $ curl -X GET http://server/database/bug
> > {"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"
> > }
> >
> > $ curl -X DELETE
> > http://server/database/bug?rev=1-59414e77c768bc202142ac82c2f129de
> > {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
> >
> > $ curl -X GET http://server/database/bug
> > {"error":"not_found","reason":"deleted"}
> >
> > $ curl -X POST -H "Content-type: application/json"
> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> > : "value" } ] }'
> > [{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
> >
> > $ curl -X GET http://server/database/bug
> > {"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"
> > }
> >
> > $ curl -X DELETE
> > http://server/database/bug?rev=3-82b9390af5b7c003e03c0dd7e6aac45a
> > {"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
> >
> > $ curl -X POST -H "Content-type: application/json"
> > http://server/database/_compact
> > {"ok":true}
> >
> > $ curl -X GET http://server/database/bug
> > {"error":"not_found","reason":"deleted"}
> >
> > $ curl -X POST -H "Content-type: application/json"
> > http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> > : "value" } ] }'
> > [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
> >
> > $ curl -X GET http://server/database/bug
> > {"error":"not_found","reason":"deleted"}
> >
> >
> > -----Original Message-----
> > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> > Sent: 23 November 2011 14:38
> > To: user@couchdb.apache.org
> > Subject: Re: possible compact bug in 1.1.1
> >
> > When a new document is created, the response is 201, not 200.
> > If the posted document is identical to the one saved, the revision is
> > the same (otherwise it would have been a conflict). So the revision
> > you have is not obsolete.
> >
> > Marcello
> >
> > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> > <Vi...@morganstanley.com>:
> > > I'd rather know about the fact that I haven't just successfully
> > > created a doc, but re-submitted a revision that was already known -
> > > and is already obsolete as revisions with higher version numbers
> > > already
> > exist.
> > >
> > > Cheers,
> > > Viktor
> > >
> > > -----Original Message-----
> > > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> > > Sent: 23 November 2011 13:41
> > > To: user@couchdb.apache.org
> > > Subject: Re: possible compact bug in 1.1.1
> > >
> > > Can you elaborate on why you want a conflict?
> > > I find it confusing to have a conflict when, in fact, there can't be
> > > any
> > conflict since nothing has changed.
> > >
> > > Marcello
> > >
> > > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> > > <Vi...@morganstanley.com>:
> > >> Thanks Paul, this makes sense.
> > >>
> > >> If it counts, I vote for forcing a conflict ;)
> > >>
> > >> Cheers,
> > >> Viktor
> > >>
> > >> -----Original Message-----
> > >> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
> > >> Sent: 22 November 2011 20:54
> > >> To: user@couchdb.apache.org
> > >> Subject: Re: possible compact bug in 1.1.1
> > >>
> > >>
> > >> Your example here is actually hitting a very specific edge case as
> > demonstrated by Marcello's test. As of many versions ago, revisions
> > are generated using a hashing scheme of the document contents. In your
> > particular case the requests you're issuing contain the same identical
> > data in such a way that CouchDB will generate a revision of the doc.
> > >>
> > >> Given this, we then have to look at how this plays into replication.
> > >> Basically, when we merge the revision trees we get to the case
> > >> where
> > it's "oh, we already have this version, cool" because we do already
> > have this version.
> > >>
> > >> Whether or not that behavior is best, or if we should force a
> > >> conflict
> > if we don't add a leaf during a write is another question. In other
> > words, the system is working fine, but this particular behavior can be
> > a bit unexpected.
> > >>
> > >> -------------------------------------------------------------------
> > >> --
> > >> -
> > >> ----
> > >> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies and notify the sender immediately. Mistransmission is not
> > intended to waive confidentiality or privilege. Morgan Stanley
> > reserves the right, to the extent permitted under applicable law, to
> > monitor electronic communications. This message is subject to terms
> > available at the following
> > link: http://www.morganstanley.com/disclaimers. If you cannot access
> > these links, please notify us by reply message and we will send the
> > contents to you. By messaging with Morgan Stanley you consent to the
> > foregoing.
> > >
> > > --------------------------------------------------------------------
> > > --
> > > ----
> > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies and notify the sender immediately. Mistransmission is not
> > intended to waive confidentiality or privilege. Morgan Stanley
> > reserves the right, to the extent permitted under applicable law, to
> > monitor electronic communications. This message is subject to terms
> > available at the following
> > link: http://www.morganstanley.com/disclaimers. If you cannot access
> > these links, please notify us by reply message and we will send the
> > contents to you. By messaging with Morgan Stanley you consent to the
> > foregoing.
> >
> > ----------------------------------------------------------------------
> > ----
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies and notify the sender immediately. Mistransmission is not
> > intended to waive confidentiality or privilege. Morgan Stanley
> > reserves the right, to the extent permitted under applicable law, to
> > monitor electronic communications. This message is subject to terms
> > available at the following
> > link: http://www.morganstanley.com/disclaimers. If you cannot access
> > these links, please notify us by reply message and we will send the
> > contents to you. By messaging with Morgan Stanley you consent to the
> > foregoing.
> >
>
> --------------------------------------------------------------------------
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies and
> notify the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to the
> extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
>
RE: possible compact bug in 1.1.1
Posted by "Szabo, Viktor" <Vi...@morganstanley.com>.
Yes, the re-insertion took place after the compaction completed.
Should it matter? Does it matter?
Thanks,
Viktor
-----Original Message-----
From: CGS [mailto:cgsmcmlxxv@gmail.com]
Sent: 20 February 2012 23:22
To: user@couchdb.apache.org
Subject: Re: possible compact bug in 1.1.1
Hi,
Was the compaction finished when you tried to insert the document?
CGS
On Mon, Feb 20, 2012 at 10:16 PM, Szabo, Viktor < Viktor.Szabo@morganstanley.com> wrote:
> Hi,
>
> An application we have is using bulk writes to insert objects into the
> database, and occasionally some documents get lost. It turns out this
> happens for the very same reason described below. If we're re-insering
> a doc using an id that has been deleted earlier, if the contents match
> the payload that was saved when the id got used for the first time,
> the doc remains in "deleted" state and the client gets a misleading
> response. This only happens when the database is compacted after the
> delete operation is executed.
>
> I still feel that this is pretty nasty - what do you guys think?
>
> Looking at the reply coming from _bulk_docs, there's no way I can tell
> if I can read back my doc from the database or not.
>
> Below are the steps to reproduce.
>
> Thanks,
> Viktor
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> : "value" } ] }'
> [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>
> $ curl -X GET http://server/database/bug
> {"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"
> }
>
> $ curl -X DELETE
> http://server/database/bug?rev=1-59414e77c768bc202142ac82c2f129de
> {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
>
> $ curl -X GET http://server/database/bug
> {"error":"not_found","reason":"deleted"}
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> : "value" } ] }'
> [{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
>
> $ curl -X GET http://server/database/bug
> {"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"
> }
>
> $ curl -X DELETE
> http://server/database/bug?rev=3-82b9390af5b7c003e03c0dd7e6aac45a
> {"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_compact
> {"ok":true}
>
> $ curl -X GET http://server/database/bug
> {"error":"not_found","reason":"deleted"}
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> : "value" } ] }'
> [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>
> $ curl -X GET http://server/database/bug
> {"error":"not_found","reason":"deleted"}
>
>
> -----Original Message-----
> From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> Sent: 23 November 2011 14:38
> To: user@couchdb.apache.org
> Subject: Re: possible compact bug in 1.1.1
>
> When a new document is created, the response is 201, not 200.
> If the posted document is identical to the one saved, the revision is
> the same (otherwise it would have been a conflict). So the revision
> you have is not obsolete.
>
> Marcello
>
> 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> <Vi...@morganstanley.com>:
> > I'd rather know about the fact that I haven't just successfully
> > created a doc, but re-submitted a revision that was already known -
> > and is already obsolete as revisions with higher version numbers
> > already
> exist.
> >
> > Cheers,
> > Viktor
> >
> > -----Original Message-----
> > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> > Sent: 23 November 2011 13:41
> > To: user@couchdb.apache.org
> > Subject: Re: possible compact bug in 1.1.1
> >
> > Can you elaborate on why you want a conflict?
> > I find it confusing to have a conflict when, in fact, there can't be
> > any
> conflict since nothing has changed.
> >
> > Marcello
> >
> > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> > <Vi...@morganstanley.com>:
> >> Thanks Paul, this makes sense.
> >>
> >> If it counts, I vote for forcing a conflict ;)
> >>
> >> Cheers,
> >> Viktor
> >>
> >> -----Original Message-----
> >> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
> >> Sent: 22 November 2011 20:54
> >> To: user@couchdb.apache.org
> >> Subject: Re: possible compact bug in 1.1.1
> >>
> >>
> >> Your example here is actually hitting a very specific edge case as
> demonstrated by Marcello's test. As of many versions ago, revisions
> are generated using a hashing scheme of the document contents. In your
> particular case the requests you're issuing contain the same identical
> data in such a way that CouchDB will generate a revision of the doc.
> >>
> >> Given this, we then have to look at how this plays into replication.
> >> Basically, when we merge the revision trees we get to the case
> >> where
> it's "oh, we already have this version, cool" because we do already
> have this version.
> >>
> >> Whether or not that behavior is best, or if we should force a
> >> conflict
> if we don't add a leaf during a write is another question. In other
> words, the system is working fine, but this particular behavior can be
> a bit unexpected.
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -
> >> ----
> >> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies and notify the sender immediately. Mistransmission is not
> intended to waive confidentiality or privilege. Morgan Stanley
> reserves the right, to the extent permitted under applicable law, to
> monitor electronic communications. This message is subject to terms
> available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
> >
> > --------------------------------------------------------------------
> > --
> > ----
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies and notify the sender immediately. Mistransmission is not
> intended to waive confidentiality or privilege. Morgan Stanley
> reserves the right, to the extent permitted under applicable law, to
> monitor electronic communications. This message is subject to terms
> available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
>
> ----------------------------------------------------------------------
> ----
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies and notify the sender immediately. Mistransmission is not
> intended to waive confidentiality or privilege. Morgan Stanley
> reserves the right, to the extent permitted under applicable law, to
> monitor electronic communications. This message is subject to terms
> available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
>
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: possible compact bug in 1.1.1
Posted by CGS <cg...@gmail.com>.
Hi,
Was the compaction finished when you tried to insert the document?
CGS
On Mon, Feb 20, 2012 at 10:16 PM, Szabo, Viktor <
Viktor.Szabo@morganstanley.com> wrote:
> Hi,
>
> An application we have is using bulk writes to insert objects into the
> database, and occasionally some
> documents get lost. It turns out this happens for the very same reason
> described below. If we're
> re-insering a doc using an id that has been deleted earlier, if the
> contents match the payload that was
> saved when the id got used for the first time, the doc remains in
> "deleted" state and the client
> gets a misleading response. This only happens when the database is
> compacted after the delete operation
> is executed.
>
> I still feel that this is pretty nasty - what do you guys think?
>
> Looking at the reply coming from _bulk_docs, there's no way I can tell if
> I can read back my doc from the database or not.
>
> Below are the steps to reproduce.
>
> Thanks,
> Viktor
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> : "value" } ] }'
> [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>
> $ curl -X GET http://server/database/bug
> {"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"}
>
> $ curl -X DELETE
> http://server/database/bug?rev=1-59414e77c768bc202142ac82c2f129de
> {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
>
> $ curl -X GET http://server/database/bug
> {"error":"not_found","reason":"deleted"}
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> : "value" } ] }'
> [{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
>
> $ curl -X GET http://server/database/bug
> {"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"}
>
> $ curl -X DELETE
> http://server/database/bug?rev=3-82b9390af5b7c003e03c0dd7e6aac45a
> {"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_compact
> {"ok":true}
>
> $ curl -X GET http://server/database/bug
> {"error":"not_found","reason":"deleted"}
>
> $ curl -X POST -H "Content-type: application/json"
> http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key"
> : "value" } ] }'
> [{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
>
> $ curl -X GET http://server/database/bug
> {"error":"not_found","reason":"deleted"}
>
>
> -----Original Message-----
> From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> Sent: 23 November 2011 14:38
> To: user@couchdb.apache.org
> Subject: Re: possible compact bug in 1.1.1
>
> When a new document is created, the response is 201, not 200.
> If the posted document is identical to the one saved, the revision is the
> same (otherwise it would have been a conflict). So the revision you have is
> not obsolete.
>
> Marcello
>
> 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> <Vi...@morganstanley.com>:
> > I'd rather know about the fact that I haven't just successfully
> > created a doc, but re-submitted a revision that was already known -
> > and is already obsolete as revisions with higher version numbers already
> exist.
> >
> > Cheers,
> > Viktor
> >
> > -----Original Message-----
> > From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> > Sent: 23 November 2011 13:41
> > To: user@couchdb.apache.org
> > Subject: Re: possible compact bug in 1.1.1
> >
> > Can you elaborate on why you want a conflict?
> > I find it confusing to have a conflict when, in fact, there can't be any
> conflict since nothing has changed.
> >
> > Marcello
> >
> > 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> > <Vi...@morganstanley.com>:
> >> Thanks Paul, this makes sense.
> >>
> >> If it counts, I vote for forcing a conflict ;)
> >>
> >> Cheers,
> >> Viktor
> >>
> >> -----Original Message-----
> >> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
> >> Sent: 22 November 2011 20:54
> >> To: user@couchdb.apache.org
> >> Subject: Re: possible compact bug in 1.1.1
> >>
> >>
> >> Your example here is actually hitting a very specific edge case as
> demonstrated by Marcello's test. As of many versions ago, revisions are
> generated using a hashing scheme of the document contents. In your
> particular case the requests you're issuing contain the same identical data
> in such a way that CouchDB will generate a revision of the doc.
> >>
> >> Given this, we then have to look at how this plays into replication.
> >> Basically, when we merge the revision trees we get to the case where
> it's "oh, we already have this version, cool" because we do already have
> this version.
> >>
> >> Whether or not that behavior is best, or if we should force a conflict
> if we don't add a leaf during a write is another question. In other words,
> the system is working fine, but this particular behavior can be a bit
> unexpected.
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> ----
> >> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies and
> notify the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to the
> extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
> >
> > ----------------------------------------------------------------------
> > ----
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies and
> notify the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to the
> extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
>
> --------------------------------------------------------------------------
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies and
> notify the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to the
> extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following
> link: http://www.morganstanley.com/disclaimers. If you cannot access
> these links, please notify us by reply message and we will send the
> contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
>
Re: possible compact bug in 1.1.1
Posted by Dave Cottlehuber <da...@muse.net.nz>.
2012/2/21 Szabo, Viktor <Vi...@morganstanley.com>:
> Hi Rogutės,
>
> Thanks, I've just created a ticket under COUCHDB-1415.
>
> I've marked it as "Major" although it could be considered "Critical" given the fact
> one might lose the data thinking it's stored into the database while it's not.
>
> Thanks,
> Viktor
Hi Viktor,
Thanks for reporting this. While I missed the discussion we are aware
of it ; more news via jira.
Cheers
Dave
RE: possible compact bug in 1.1.1
Posted by "Szabo, Viktor" <Vi...@morganstanley.com>.
Hi Rogutės,
Thanks, I've just created a ticket under COUCHDB-1415.
I've marked it as "Major" although it could be considered "Critical" given the fact
one might lose the data thinking it's stored into the database while it's not.
Thanks,
Viktor
-----Original Message-----
From: Rogutės Sparnuotos [mailto:rogutes@googlemail.com]
Sent: 21 February 2012 14:16
To: user@couchdb.apache.org
Subject: Re: possible compact bug in 1.1.1
Szabo, Viktor (2012-02-20 21:16):
> Hi,
>
> An application we have is using bulk writes to insert objects into the
> database, and occasionally some documents get lost. It turns out this
> happens for the very same reason described below. If we're re-insering
> a doc using an id that has been deleted earlier, if the contents match
> the payload that was saved when the id got used for the first time,
> the doc remains in "deleted" state and the client gets a misleading response. This only happens when the database is compacted after the delete operation is executed.
>
> I still feel that this is pretty nasty - what do you guys think?
>
> Looking at the reply coming from _bulk_docs, there's no way I can tell if I can read back my doc from the database or not.
This is a bug and it would be nice if you could report it to JIRA. I can also reproduce it with the latest CouchDB version from git (and this seems not really related to _bulk_docs):
alias curl='curl -H "Content-Type: application/json"'
url="http://localhost:5984/database"
1 curl -X PUT $url
2 curl -X POST $url -d '{"_id": "bug", "key": "value"}'
3 curl -X DELETE "$url/bug?rev=1-59414e77c768bc202142ac82c2f129de"
4 curl -X POST "$url/_compact"
5 curl -X POST $url -d '{"_id": "bug", "key": "value"}'
6 curl -X GET "$url/bug"
(bug here)
1 {"ok":true}
201
2 [{"ok":true,"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
201
3 {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
200
4 {"ok":true}
202
5 [{"ok":true,"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
201
6 {"error":"not_found","reason":"deleted"}
404
CouchDB shouldn't report "ok" on step 5 and then go on to claim that the doc is deleted. Also, it seems to work on second try:
7 curl -X POST $url -d '{"_id": "bug", "key": "value"}'
8 curl -X GET "$url/bug"
7 {"ok":true,"id":"bug","rev":"3-674f864b73df1c80925e48436e21d550"}
201
8 {"_id":"bug","_rev":"3-674f864b73df1c80925e48436e21d550","key":"value"}
200
--
-- Rogutės Sparnuotos
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: possible compact bug in 1.1.1
Posted by Rogutės Sparnuotos <ro...@googlemail.com>.
Szabo, Viktor (2012-02-20 21:16):
> Hi,
>
> An application we have is using bulk writes to insert objects into the database, and occasionally some
> documents get lost. It turns out this happens for the very same reason described below. If we're
> re-insering a doc using an id that has been deleted earlier, if the contents match the payload that was
> saved when the id got used for the first time, the doc remains in "deleted" state and the client
> gets a misleading response. This only happens when the database is compacted after the delete operation
> is executed.
>
> I still feel that this is pretty nasty - what do you guys think?
>
> Looking at the reply coming from _bulk_docs, there's no way I can tell if I can read back my doc from the database or not.
This is a bug and it would be nice if you could report it to JIRA. I can
also reproduce it with the latest CouchDB version from git (and this seems
not really related to _bulk_docs):
alias curl='curl -H "Content-Type: application/json"'
url="http://localhost:5984/database"
1 curl -X PUT $url
2 curl -X POST $url -d '{"_id": "bug", "key": "value"}'
3 curl -X DELETE "$url/bug?rev=1-59414e77c768bc202142ac82c2f129de"
4 curl -X POST "$url/_compact"
5 curl -X POST $url -d '{"_id": "bug", "key": "value"}'
6 curl -X GET "$url/bug"
(bug here)
1 {"ok":true}
201
2 [{"ok":true,"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
201
3 {"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
200
4 {"ok":true}
202
5 [{"ok":true,"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
201
6 {"error":"not_found","reason":"deleted"}
404
CouchDB shouldn't report "ok" on step 5 and then go on to claim that the
doc is deleted. Also, it seems to work on second try:
7 curl -X POST $url -d '{"_id": "bug", "key": "value"}'
8 curl -X GET "$url/bug"
7 {"ok":true,"id":"bug","rev":"3-674f864b73df1c80925e48436e21d550"}
201
8 {"_id":"bug","_rev":"3-674f864b73df1c80925e48436e21d550","key":"value"}
200
--
-- Rogutės Sparnuotos