You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Mark Hammond <sk...@gmail.com> on 2010/02/19 04:29:50 UTC
include_docs returning doc=null
I've struck a rare case where a view request with include_docs=true (but
no stale=ok and no custom _id emitted from the view) is returning a row
with doc=null.
I guess is might be possible for a document to be deleted between the
view row being processed and the doc being opened for the view result
but I don't think this is happening here - the view also emits the
document _rev, and directly after getting doc=null I can open the
specified document in futon and it has the same _rev as the view row.
Unfortunately, couch is not offering any clues about what the problem is.
I tracked the code returning null to couch_httpd_view:doc_member - it does:
case (catch couch_httpd_db:couch_doc_open(Db, DocId, Rev, [])) of
#doc{} = Doc ->
JsonDoc = couch_doc:to_json_obj(Doc, []),
[{doc, JsonDoc}];
_Else ->
[{doc, null}]
So should the doc fail to open for any reason, you just get a null and
the reason for the failure is discarded. Is this by intent? ISTM it
would be better to instead, or also, provide an {error, Reason} element
in the view row.
Does this sound desirable? If so, should an error still leave a
doc=null element, or only include the error element? Let me know and
I'll open a but and submit a patch...
Cheers,
Mark
Re: include_docs returning doc=null
Posted by Mark Hammond <mh...@skippinet.com.au>.
On 20/02/2010 3:48 PM, Paul Davis wrote:
> Sudden thought, are you doing something like "save doc, read view"
> real real quick?
Yep. There is a bit of concurrency going on to - so it is possible the
view is being read before the write of the document completes in the
client (it is a Python-twisted application - but just think multiple
independent requests rather than 'sequential' operations)
> Are we hitting a thing where a doc's committed status
> is being reflected different between the view generation and the view
> read?
I guess so...
Cheers,
Mark
Re: include_docs returning doc=null
Posted by Paul Davis <pa...@gmail.com>.
Sudden thought, are you doing something like "save doc, read view"
real real quick? Are we hitting a thing where a doc's committed status
is being reflected different between the view generation and the view
read?
I can't seem to remember the necessary details on all the related code
paths, but it seems like there might be a small window there that
something is a tidge wonky.
On Fri, Feb 19, 2010 at 11:17 PM, Mark Hammond <sk...@gmail.com> wrote:
> A little more info. I change couch to include an 'error' element when this
> happens (see COUCHDB-662 for the patch), and bumped the log-level to debug,
> then reproduced the problem.
>
> My Python code which occasionally triggers this problem reports the returned
> row as:
>
> {u'value':
> {u'_rev': u'1-6ace70202b3eba6f7a7190d4af65b918', ...},
> u'error': u'not_found',
> u'id':
> u'rc!email.IjAwMTYzNjQ2Y2MyODI3OTBiMzA0NmZiYzJhYzdAZ29vZ2xlbWFpbC5jb20i!rd.msg.conversation',
> u'key': [...],
> u'doc': None
> }
>
> Note the new 'error' element. Immediately after this happens, I open the
> doc in futon and see the document exists with a _rev of
> 1-6ace70202b3eba6f7a7190d4af65b918 (ie, exactly as above). It seems it
> should be 'impossible' for things to get into this state - the doc existed
> when the view was updated, and seems to exist in that same form now.
>
> The couch log reports:
>
> [Sat, 20 Feb 2010 03:56:18 GMT] [debug] [<0.2076.0>] Include Doc:
> <<"rc!email.IjAwMTYzNjQ2Y2MyODI3OTBiMzA0NmZiYzJhYzdAZ29vZ2xlbWFpbC5jb20i!rd.msg.conversation">>
> nil
>
> So as we suspected, when couch is including the documents for a view, it is
> asking for the latest revision - but as mentioned above, the latest revision
> seems to be the same revision the view was created with.
>
> Is there anything else I can do to help get to the bottom of this?
>
> Thanks,
>
> Mark
>
Re: include_docs returning doc=null
Posted by Mark Hammond <sk...@gmail.com>.
A little more info. I change couch to include an 'error' element when
this happens (see COUCHDB-662 for the patch), and bumped the log-level
to debug, then reproduced the problem.
My Python code which occasionally triggers this problem reports the
returned row as:
{u'value':
{u'_rev': u'1-6ace70202b3eba6f7a7190d4af65b918', ...},
u'error': u'not_found',
u'id':
u'rc!email.IjAwMTYzNjQ2Y2MyODI3OTBiMzA0NmZiYzJhYzdAZ29vZ2xlbWFpbC5jb20i!rd.msg.conversation',
u'key': [...],
u'doc': None
}
Note the new 'error' element. Immediately after this happens, I open
the doc in futon and see the document exists with a _rev of
1-6ace70202b3eba6f7a7190d4af65b918 (ie, exactly as above). It seems it
should be 'impossible' for things to get into this state - the doc
existed when the view was updated, and seems to exist in that same form now.
The couch log reports:
[Sat, 20 Feb 2010 03:56:18 GMT] [debug] [<0.2076.0>] Include Doc:
<<"rc!email.IjAwMTYzNjQ2Y2MyODI3OTBiMzA0NmZiYzJhYzdAZ29vZ2xlbWFpbC5jb20i!rd.msg.conversation">>
nil
So as we suspected, when couch is including the documents for a view, it
is asking for the latest revision - but as mentioned above, the latest
revision seems to be the same revision the view was created with.
Is there anything else I can do to help get to the bottom of this?
Thanks,
Mark
Re: include_docs returning doc=null
Posted by Mark Hammond <sk...@gmail.com>.
Brian writes:
> On Fri, Feb 19, 2010 at 02:29:50PM +1100, Mark Hammond wrote:
>> I guess is might be possible for a document to be deleted between
>> the view row being processed and the doc being opened for the view
>> result but I don't think this is happening here - the view also
>> emits the document _rev
>
> I didn't think the view Btree index included the _rev. If it did it would
> allow really neat things like proper view etags, as well as preventing the
> race condition you describe.
>
> However if it did, I would also expect the rev to be in the raw view rows,
> i.e.
>
> {"id":"xxx","rev":"yyy","key":"aaa","value":"bbb","doc":{...}}
> ^^^^^^^^^^^
>
> Do you mean that you have emit(..., {_rev: doc._rev}) in your view?
Correct.
>> I tracked the code returning null to couch_httpd_view:doc_member - it does:
>>
>> case (catch couch_httpd_db:couch_doc_open(Db, DocId, Rev, [])) of
>
> There is a LOG_DEBUG just above that. What does it show? In particular, is
> Rev nil? In that case you'll just get "current" version. Or is it getting
> the rev of a deleted doc?
I came to the conclusion couch had a null rev, so is attempting to get
the latest. I haven't seen this happen with debug logs - it happens
fairly rarely. I did add another log for that error case and saw couch
is reporting the error as not_found.
Adam writes:
> Mark, is it fully reproducible? You can always read the document directly, but never through include_docs=true?
It isn't "fully" reproducible as such, but a test suite I have will
strike this on one particular test rougly 5-10% of the times it is run.
I'll try logging what _rev couch is attempting to use...
Cheers,
Mark
Re: include_docs returning doc=null
Posted by Brian Candler <B....@pobox.com>.
On Fri, Feb 19, 2010 at 02:29:50PM +1100, Mark Hammond wrote:
> I guess is might be possible for a document to be deleted between
> the view row being processed and the doc being opened for the view
> result but I don't think this is happening here - the view also
> emits the document _rev
I didn't think the view Btree index included the _rev. If it did it would
allow really neat things like proper view etags, as well as preventing the
race condition you describe.
However if it did, I would also expect the rev to be in the raw view rows,
i.e.
{"id":"xxx","rev":"yyy","key":"aaa","value":"bbb","doc":{...}}
^^^^^^^^^^^
Do you mean that you have emit(..., {_rev: doc._rev}) in your view? In that
case check the code to see if the _rev is used without an explicit _id. Or
you could try
emit(..., {_id: doc._id, _rev: doc._rev})
to see if that behaves any better.
> I tracked the code returning null to couch_httpd_view:doc_member - it does:
>
> case (catch couch_httpd_db:couch_doc_open(Db, DocId, Rev, [])) of
There is a LOG_DEBUG just above that. What does it show? In particular, is
Rev nil? In that case you'll just get "current" version. Or is it getting
the rev of a deleted doc?
I tried walking the code but got a bit lost. I was trying to check whether
Rev is always nil, or is taken from view row _rev member, or somewhere else.
Regards,
Brian.
Re: include_docs returning doc=null
Posted by Adam Kocoloski <ko...@apache.org>.
Indeed we did. One of our customers bumped into a similar failure mode the other day too. In his case the doc member returned was {error, missing}, which pretty much completely baffles me. Mark's example is also baffling because it does not use stale=ok, so _any_ error opening the document is troubling.
We have code in couch_view_group that ostensibly prevents the view index header from being saved if the DB committed_update_seq is still behind it. We do that precisely to prevent this situation from happening. We should also make it so that the version of the DB used to satisfy the include_docs request is the same MVCC snapshot used by the view group. I think we already do.
I guess another alternative to a logic error in couch_view_group is in-flight corruption. If the MD5 checksum of what we read back from disk doesn't match the stored MD5 we do throw(file_corruption). But the catch statement in that pasted code hides the error condition from the client. I'm definitely +1 on an {error, Reason} tuple there. That call should never fail.
Mark, is it fully reproducible? You can always read the document directly, but never through include_docs=true?
Adam
On Feb 19, 2010, at 8:14 AM, Paul Davis wrote:
> I'd be +1 returning an error. In fact we used to IIRC.
>
> Paul
>
> On Thu, Feb 18, 2010 at 10:29 PM, Mark Hammond <sk...@gmail.com> wrote:
>> I've struck a rare case where a view request with include_docs=true (but no
>> stale=ok and no custom _id emitted from the view) is returning a row with
>> doc=null.
>>
>> I guess is might be possible for a document to be deleted between the view
>> row being processed and the doc being opened for the view result but I don't
>> think this is happening here - the view also emits the document _rev, and
>> directly after getting doc=null I can open the specified document in futon
>> and it has the same _rev as the view row. Unfortunately, couch is not
>> offering any clues about what the problem is.
>>
>> I tracked the code returning null to couch_httpd_view:doc_member - it does:
>>
>> case (catch couch_httpd_db:couch_doc_open(Db, DocId, Rev, [])) of
>> #doc{} = Doc ->
>> JsonDoc = couch_doc:to_json_obj(Doc, []),
>> [{doc, JsonDoc}];
>> _Else ->
>> [{doc, null}]
>>
>> So should the doc fail to open for any reason, you just get a null and the
>> reason for the failure is discarded. Is this by intent? ISTM it would be
>> better to instead, or also, provide an {error, Reason} element in the view
>> row.
>>
>> Does this sound desirable? If so, should an error still leave a doc=null
>> element, or only include the error element? Let me know and I'll open a but
>> and submit a patch...
>>
>> Cheers,
>>
>> Mark
>>
Re: include_docs returning doc=null
Posted by Paul Davis <pa...@gmail.com>.
I'd be +1 returning an error. In fact we used to IIRC.
Paul
On Thu, Feb 18, 2010 at 10:29 PM, Mark Hammond <sk...@gmail.com> wrote:
> I've struck a rare case where a view request with include_docs=true (but no
> stale=ok and no custom _id emitted from the view) is returning a row with
> doc=null.
>
> I guess is might be possible for a document to be deleted between the view
> row being processed and the doc being opened for the view result but I don't
> think this is happening here - the view also emits the document _rev, and
> directly after getting doc=null I can open the specified document in futon
> and it has the same _rev as the view row. Unfortunately, couch is not
> offering any clues about what the problem is.
>
> I tracked the code returning null to couch_httpd_view:doc_member - it does:
>
> case (catch couch_httpd_db:couch_doc_open(Db, DocId, Rev, [])) of
> #doc{} = Doc ->
> JsonDoc = couch_doc:to_json_obj(Doc, []),
> [{doc, JsonDoc}];
> _Else ->
> [{doc, null}]
>
> So should the doc fail to open for any reason, you just get a null and the
> reason for the failure is discarded. Is this by intent? ISTM it would be
> better to instead, or also, provide an {error, Reason} element in the view
> row.
>
> Does this sound desirable? If so, should an error still leave a doc=null
> element, or only include the error element? Let me know and I'll open a but
> and submit a patch...
>
> Cheers,
>
> Mark
>