You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@couchdb.apache.org by Marcello Nuccio <ma...@gmail.com> on 2011/11/15 12:10:51 UTC

View row with doc==null

Don't know how, but I have a view that returns the following row:

    {"id":"510427_2011_1","key":["510427",2011,1],"value":1,"doc":null}

i.e. it has doc==null when queried with include_docs=true. The doc
with the given _id is deleted. The database is compacted. The views
are compacted and cleaned (done via Futon). CouchDB is version 1.1.1
built with build-couchdb on Ubuntu 11.10.

Is this normal behaviour?

Thanks,
  Marcello

Re: View row with doc==null

Posted by Adam Kocoloski <ko...@apache.org>.

You almost certainly had a view with duplicates (caused by e.g. COUCHDB-968 or COUCHDB-1265).  When you deleted the document it left some orphaned rows in the view.  Unfortunately, the only way to remove them at present is to compact the database and then rebuild the view from scratch.

To dev@: on a quick read of the source code it looks to me like couch_httpd_view:doc_member/3 ought to be adding the 'deleted' atom to the Options in the second clause.  It's not a huge bug, but I think it would be more informative to the user to show that the doc has been deleted instead of just reporting null.  Granted, it's probably only applicable for this case of dupes in the index now that _changes loads documents directly from the #doc_info directly (the first clause handles #doc_infos and does add 'deleted' to the Options).

Best, Adam

On Nov 16, 2011, at 2:17 AM, Victor Nicollet wrote:

> I have recently encountered this problem as well on a 1.1.0 production
> database. Like you say, the "fix" was to rebuild the view. I do not know
> the reason for this behavior, but I believe it would be important to
> investigate.
> 
> On 16 November 2011 08:13, Marcello Nuccio <ma...@gmail.com>wrote:
> 
>> No, the row is:
>> 
>>  {"id":"510427_2011_1","key":["510427",2011,1],"value":1,"doc":null}
>> 
>> Marcello
>> 
>> 2011/11/16 Mark Hahn <ma...@hahnca.com>:
>>> Does this doc by any chance have an empty _id field?  If so that is a
>> known
>>> problem.  It cannot be fixed in a db when it happens.  Couch is supposed
>> to
>>> have a fix soon, if not already.
>>> 
>>> On Tue, Nov 15, 2011 at 10:57 PM, Marcello Nuccio <
>> marcello.nuccio@gmail.com
>>>> wrote:
>>> 
>>>> I've reloaded Futon at least three times...
>>>> 
>>>> However, the problem is that the row with doc==null is still there,
>>>> and the only way to get rid of it is to rebuild the view. I will try
>>>> to find how to reproduce it in a new database.
>>>> 
>>>> If anyone is interested to see the database, I can send it. It does
>>>> not contain top secret data, and it's only a couple of MB.
>>>> 
>>>> I have updated my program to ignore those rows. It is a good thing to
>>>> do since it can happen if querying with stale=ok.
>>>> 
>>>> Marcello
>>>> 
>>>> 2011/11/15 Mark Hahn <ma...@hahnca.com>:
>>>>> You have to reload twice to update futon.  It is a pita.
>>>>> 
>>>>> On Tue, Nov 15, 2011 at 5:32 AM, Marcello Nuccio
>>>>> <ma...@gmail.com>wrote:
>>>>> 
>>>>>> 2011/11/15 Marcello Nuccio <ma...@gmail.com>:
>>>>>>> I've done some more testing:
>>>>>>> 
>>>>>>> 1. I've restored the original db and view files (before compaction
>> and
>>>>>> cleanup)
>>>>>>> 2. I've made a clone of the db and view with new name
>>>>>>> 3. Verified that everything works as expected, i.e. both db return
>>>>>>> identical results and all files are identical (done using curl and
>>>>>>> cmp).
>>>>>>> 
>>>>>>> Here the tests:
>>>>>>> 
>>>>>>> 1. Compacting the clone the row with doc==null disappears.
>>>>>>> 2. Compacting the master the row is still here.
>>>>>> 
>>>>>> Ops sorry, I did those two tests with Futon and forcing reload did
>> not
>>>>>> show the row in test 1. After clearing the cache completely, the row
>>>>>> with doc==null is here.
>>>>>> 
>>>>>> I verified with curl and both dbs return the row with doc==null.
>>>>>> 
>>>>>> The only way I have found to delete it, is to delete the view file.
>>>>>> 
>>>>>> Marcello
>>>>>> 
>>>>>> 
>>>>>>> 3. Comparing with cmp *.couch and *.view, they are identical.
>>>>>>> 4. Restarting couchdb does not help.
>>>>>>> 
>>>>>>> What am I missing?
>>>>>>> 
>>>>>>> Marcello
>>>>>>> 
>>>>>>> 2011/11/15 Robert Newson <rn...@apache.org>:
>>>>>>>> That does sound wrong but not sure what steps to suggest next.
>>>>>>>> 
>>>>>>>> B.
>>>>>>>> 
>>>>>>>> On 15 November 2011 12:22, Marcello Nuccio <
>>>> marcello.nuccio@gmail.com>
>>>>>> wrote:
>>>>>>>>> No.
>>>>>>>>> 
>>>>>>>>> Marcello
>>>>>>>>> 
>>>>>>>>> 2011/11/15 Robert Newson <rn...@apache.org>:
>>>>>>>>>> Did you query with stale=ok?
>>>>>>>>>> 
>>>>>>>>>> B.
>>>>>>>>>> 
>>>>>>>>>> On 15 November 2011 11:10, Marcello Nuccio <
>>>> marcello.nuccio@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> Don't know how, but I have a view that returns the following
>> row:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>> {"id":"510427_2011_1","key":["510427",2011,1],"value":1,"doc":null}
>>>>>>>>>>> 
>>>>>>>>>>> i.e. it has doc==null when queried with include_docs=true. The
>> doc
>>>>>>>>>>> with the given _id is deleted. The database is compacted. The
>>>> views
>>>>>>>>>>> are compacted and cleaned (done via Futon). CouchDB is version
>>>> 1.1.1
>>>>>>>>>>> built with build-couchdb on Ubuntu 11.10.
>>>>>>>>>>> 
>>>>>>>>>>> Is this normal behaviour?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Marcello
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Victor Nicollet, Directeur Technique, www.runorg.com

Re: View row with doc==null

Posted by Adam Kocoloski <ko...@apache.org>.

You almost certainly had a view with duplicates (caused by e.g. COUCHDB-968 or COUCHDB-1265).  When you deleted the document it left some orphaned rows in the view.  Unfortunately, the only way to remove them at present is to compact the database and then rebuild the view from scratch.

To dev@: on a quick read of the source code it looks to me like couch_httpd_view:doc_member/3 ought to be adding the 'deleted' atom to the Options in the second clause.  It's not a huge bug, but I think it would be more informative to the user to show that the doc has been deleted instead of just reporting null.  Granted, it's probably only applicable for this case of dupes in the index now that _changes loads documents directly from the #doc_info directly (the first clause handles #doc_infos and does add 'deleted' to the Options).

Best, Adam

On Nov 16, 2011, at 2:17 AM, Victor Nicollet wrote:

> I have recently encountered this problem as well on a 1.1.0 production
> database. Like you say, the "fix" was to rebuild the view. I do not know
> the reason for this behavior, but I believe it would be important to
> investigate.
> 
> On 16 November 2011 08:13, Marcello Nuccio <ma...@gmail.com>wrote:
> 
>> No, the row is:
>> 
>>  {"id":"510427_2011_1","key":["510427",2011,1],"value":1,"doc":null}
>> 
>> Marcello
>> 
>> 2011/11/16 Mark Hahn <ma...@hahnca.com>:
>>> Does this doc by any chance have an empty _id field?  If so that is a
>> known
>>> problem.  It cannot be fixed in a db when it happens.  Couch is supposed
>> to
>>> have a fix soon, if not already.
>>> 
>>> On Tue, Nov 15, 2011 at 10:57 PM, Marcello Nuccio <
>> marcello.nuccio@gmail.com
>>>> wrote:
>>> 
>>>> I've reloaded Futon at least three times...
>>>> 
>>>> However, the problem is that the row with doc==null is still there,
>>>> and the only way to get rid of it is to rebuild the view. I will try
>>>> to find how to reproduce it in a new database.
>>>> 
>>>> If anyone is interested to see the database, I can send it. It does
>>>> not contain top secret data, and it's only a couple of MB.
>>>> 
>>>> I have updated my program to ignore those rows. It is a good thing to
>>>> do since it can happen if querying with stale=ok.
>>>> 
>>>> Marcello
>>>> 
>>>> 2011/11/15 Mark Hahn <ma...@hahnca.com>:
>>>>> You have to reload twice to update futon.  It is a pita.
>>>>> 
>>>>> On Tue, Nov 15, 2011 at 5:32 AM, Marcello Nuccio
>>>>> <ma...@gmail.com>wrote:
>>>>> 
>>>>>> 2011/11/15 Marcello Nuccio <ma...@gmail.com>:
>>>>>>> I've done some more testing:
>>>>>>> 
>>>>>>> 1. I've restored the original db and view files (before compaction
>> and
>>>>>> cleanup)
>>>>>>> 2. I've made a clone of the db and view with new name
>>>>>>> 3. Verified that everything works as expected, i.e. both db return
>>>>>>> identical results and all files are identical (done using curl and
>>>>>>> cmp).
>>>>>>> 
>>>>>>> Here the tests:
>>>>>>> 
>>>>>>> 1. Compacting the clone the row with doc==null disappears.
>>>>>>> 2. Compacting the master the row is still here.
>>>>>> 
>>>>>> Ops sorry, I did those two tests with Futon and forcing reload did
>> not
>>>>>> show the row in test 1. After clearing the cache completely, the row
>>>>>> with doc==null is here.
>>>>>> 
>>>>>> I verified with curl and both dbs return the row with doc==null.
>>>>>> 
>>>>>> The only way I have found to delete it, is to delete the view file.
>>>>>> 
>>>>>> Marcello
>>>>>> 
>>>>>> 
>>>>>>> 3. Comparing with cmp *.couch and *.view, they are identical.
>>>>>>> 4. Restarting couchdb does not help.
>>>>>>> 
>>>>>>> What am I missing?
>>>>>>> 
>>>>>>> Marcello
>>>>>>> 
>>>>>>> 2011/11/15 Robert Newson <rn...@apache.org>:
>>>>>>>> That does sound wrong but not sure what steps to suggest next.
>>>>>>>> 
>>>>>>>> B.
>>>>>>>> 
>>>>>>>> On 15 November 2011 12:22, Marcello Nuccio <
>>>> marcello.nuccio@gmail.com>
>>>>>> wrote:
>>>>>>>>> No.
>>>>>>>>> 
>>>>>>>>> Marcello
>>>>>>>>> 
>>>>>>>>> 2011/11/15 Robert Newson <rn...@apache.org>:
>>>>>>>>>> Did you query with stale=ok?
>>>>>>>>>> 
>>>>>>>>>> B.
>>>>>>>>>> 
>>>>>>>>>> On 15 November 2011 11:10, Marcello Nuccio <
>>>> marcello.nuccio@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> Don't know how, but I have a view that returns the following
>> row:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>> {"id":"510427_2011_1","key":["510427",2011,1],"value":1,"doc":null}
>>>>>>>>>>> 
>>>>>>>>>>> i.e. it has doc==null when queried with include_docs=true. The
>> doc
>>>>>>>>>>> with the given _id is deleted. The database is compacted. The
>>>> views
>>>>>>>>>>> are compacted and cleaned (done via Futon). CouchDB is version
>>>> 1.1.1
>>>>>>>>>>> built with build-couchdb on Ubuntu 11.10.
>>>>>>>>>>> 
>>>>>>>>>>> Is this normal behaviour?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Marcello
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Victor Nicollet, Directeur Technique, www.runorg.com