You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by René Brüntrup <br...@imfl.de> on 2011/06/03 15:21:27 UTC
Data loss in CouchDB 1.0.1
Dear CouchDB Community,
After restarting our productive Ubuntu 10.10 server running the official
Ubuntu CouchDB 1.0.1 package we observed a data loss in our database.
Before restarting the server the application using the database worked
without any problems. After restarting the server some recent
modifications were missing from the system and we started investigating
the problems:
The last change time of the CouchDB database file dated back about two
months. A hex dump of the CouchDB file showed no changes after this
point of time. Regular database backups that where created by
replicating into another database contained more recent data, just the
main database seemed to have stopped recording changes to disc after
2011-08-03. All database files are stored on the same encrypted LVM
volume.
The problem sounds a lot like writer process bug in 1.0.0: All data
seems to be kept in RAM, but never written to disc. However, we verified
that Ubuntu uses the patched code and the conditions that could cause
the problems in version 1.0.0 do not even occur in our system.
So far we were not able to reproduce the behavior.
Thank you for reading and we appreciate any help.
Kind Regards
René Brüntrup
Re: Data loss in CouchDB 1.0.1
Posted by "Paul J. Davis" <pa...@gmail.com>.
Hah! Also, even better tool. I misread disk for couch file the first time. And by better I mean even awesomer hack. In this case with the encrypted lvm I'd be morbidly curious what it turns up.
On Jun 6, 2011, at 10:06 PM, Jason Smith <jh...@iriscouch.com> wrote:
> Thanks! I've used good tools. I would classify grep_couch as a "bad
> tool" but maybe "best of class" :)
>
> Compaction creates a new file on the fs and unlinks the old one. If
> you compact regularly, you'll tend to have *lots* of ejson fragments
> laying around in the un-(re)-allocated free parts of the fs.
>
> That reminds me, the tool does not account for filesystem
> fragmentation or anything that would make the doc not physically
> contiguous on disk. Fortunately most docs are small (even in ejson!)
> and they survive, especially if you've compacted a few times and you
> have duplicate data on-disk but with dissimilar fragment locations.
>
> On Tue, Jun 7, 2011 at 8:38 AM, Paul J. Davis
> <pa...@gmail.com> wrote:
>> Jason,
>>
>> Good tool, but unless I'm mistaken, the issue here I that the data just doesn't exist on disk. I think we're fairly sure that this isn't the 1.0.0 bug but something else. I'm leaning towards something config specific but all of our theories appear to be incorrect given the reported observations.
>>
>> On Jun 6, 2011, at 9:29 PM, Jason Smith <jh...@iriscouch.com> wrote:
>>
>>> I once made a very simple CouchDB undelete tool. It scans your disk
>>> device for anything that looks like the on-disk CouchDB JSON format.
>>>
>>> https://github.com/jhs/grep_couch
>>>
>>> I've recovered data with it, but notably, the _id and _rev are *not*
>>> stored with the rest of a document, so you tend to get lots of docs
>>> with no _id field. (I'm considering always having an "id" field to
>>> dupe the "_id" in case I ever have to do that again.)
>>>
>>> On Mon, Jun 6, 2011 at 3:35 PM, René Brüntrup <br...@imfl.de> wrote:
>>>> Hello!
>>>>
>>>>> 1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
>>>>
>>>> We could not find the missing data in any of the replicated files. Each
>>>> backup has an increasing number of documents with the least amount of
>>>> missing data in the newest backup. Access logs from the 2011-03-08 are
>>>> unfortunately missing, because the log rotation already removed them.
>>>>
>>>>> 2) Is it possible that you've inadvertently restored the database file from a backup?
>>>>
>>>> No backup was created at this date and we do not have any mechanisms
>>>> that could automatically restore the backups.
>>>>
>>>>> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
>>>>
>>>> Our system does not work without an initialized database that contains a
>>>> number of user account and definition documents. But even if such an
>>>> initialized database would have been available underneath the encrypted
>>>> volume we would have noticed a data loss after changing the database,
>>>> because the system was already in use before the 2011-03-08.
>>>>
>>>>
>>>> We will check again, if there is a database file underneath the
>>>> encrypted volume, but we cannot stop the system right now. When
>>>> replicating the database we just noticed, that the timestamp of the
>>>> source database was updated. The replication processes that where
>>>> started after the 2011-03-08 and before the reboot of the server did not
>>>> change the timestamp of the source database.
>>>>
>>>>
>>>> Regards,
>>>> René
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Iris Couch
>>
>
>
>
> --
> Iris Couch
Re: Data loss in CouchDB 1.0.1
Posted by Jason Smith <jh...@iriscouch.com>.
Thanks! I've used good tools. I would classify grep_couch as a "bad
tool" but maybe "best of class" :)
Compaction creates a new file on the fs and unlinks the old one. If
you compact regularly, you'll tend to have *lots* of ejson fragments
laying around in the un-(re)-allocated free parts of the fs.
That reminds me, the tool does not account for filesystem
fragmentation or anything that would make the doc not physically
contiguous on disk. Fortunately most docs are small (even in ejson!)
and they survive, especially if you've compacted a few times and you
have duplicate data on-disk but with dissimilar fragment locations.
On Tue, Jun 7, 2011 at 8:38 AM, Paul J. Davis
<pa...@gmail.com> wrote:
> Jason,
>
> Good tool, but unless I'm mistaken, the issue here I that the data just doesn't exist on disk. I think we're fairly sure that this isn't the 1.0.0 bug but something else. I'm leaning towards something config specific but all of our theories appear to be incorrect given the reported observations.
>
> On Jun 6, 2011, at 9:29 PM, Jason Smith <jh...@iriscouch.com> wrote:
>
>> I once made a very simple CouchDB undelete tool. It scans your disk
>> device for anything that looks like the on-disk CouchDB JSON format.
>>
>> https://github.com/jhs/grep_couch
>>
>> I've recovered data with it, but notably, the _id and _rev are *not*
>> stored with the rest of a document, so you tend to get lots of docs
>> with no _id field. (I'm considering always having an "id" field to
>> dupe the "_id" in case I ever have to do that again.)
>>
>> On Mon, Jun 6, 2011 at 3:35 PM, René Brüntrup <br...@imfl.de> wrote:
>>> Hello!
>>>
>>>> 1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
>>>
>>> We could not find the missing data in any of the replicated files. Each
>>> backup has an increasing number of documents with the least amount of
>>> missing data in the newest backup. Access logs from the 2011-03-08 are
>>> unfortunately missing, because the log rotation already removed them.
>>>
>>>> 2) Is it possible that you've inadvertently restored the database file from a backup?
>>>
>>> No backup was created at this date and we do not have any mechanisms
>>> that could automatically restore the backups.
>>>
>>>> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
>>>
>>> Our system does not work without an initialized database that contains a
>>> number of user account and definition documents. But even if such an
>>> initialized database would have been available underneath the encrypted
>>> volume we would have noticed a data loss after changing the database,
>>> because the system was already in use before the 2011-03-08.
>>>
>>>
>>> We will check again, if there is a database file underneath the
>>> encrypted volume, but we cannot stop the system right now. When
>>> replicating the database we just noticed, that the timestamp of the
>>> source database was updated. The replication processes that where
>>> started after the 2011-03-08 and before the reboot of the server did not
>>> change the timestamp of the source database.
>>>
>>>
>>> Regards,
>>> René
>>>
>>>
>>>
>>
>>
>>
>> --
>> Iris Couch
>
--
Iris Couch
Re: Data loss in CouchDB 1.0.1
Posted by "Paul J. Davis" <pa...@gmail.com>.
Jason,
Good tool, but unless I'm mistaken, the issue here I that the data just doesn't exist on disk. I think we're fairly sure that this isn't the 1.0.0 bug but something else. I'm leaning towards something config specific but all of our theories appear to be incorrect given the reported observations.
On Jun 6, 2011, at 9:29 PM, Jason Smith <jh...@iriscouch.com> wrote:
> I once made a very simple CouchDB undelete tool. It scans your disk
> device for anything that looks like the on-disk CouchDB JSON format.
>
> https://github.com/jhs/grep_couch
>
> I've recovered data with it, but notably, the _id and _rev are *not*
> stored with the rest of a document, so you tend to get lots of docs
> with no _id field. (I'm considering always having an "id" field to
> dupe the "_id" in case I ever have to do that again.)
>
> On Mon, Jun 6, 2011 at 3:35 PM, René Brüntrup <br...@imfl.de> wrote:
>> Hello!
>>
>>> 1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
>>
>> We could not find the missing data in any of the replicated files. Each
>> backup has an increasing number of documents with the least amount of
>> missing data in the newest backup. Access logs from the 2011-03-08 are
>> unfortunately missing, because the log rotation already removed them.
>>
>>> 2) Is it possible that you've inadvertently restored the database file from a backup?
>>
>> No backup was created at this date and we do not have any mechanisms
>> that could automatically restore the backups.
>>
>>> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
>>
>> Our system does not work without an initialized database that contains a
>> number of user account and definition documents. But even if such an
>> initialized database would have been available underneath the encrypted
>> volume we would have noticed a data loss after changing the database,
>> because the system was already in use before the 2011-03-08.
>>
>>
>> We will check again, if there is a database file underneath the
>> encrypted volume, but we cannot stop the system right now. When
>> replicating the database we just noticed, that the timestamp of the
>> source database was updated. The replication processes that where
>> started after the 2011-03-08 and before the reboot of the server did not
>> change the timestamp of the source database.
>>
>>
>> Regards,
>> René
>>
>>
>>
>
>
>
> --
> Iris Couch
Re: Data loss in CouchDB 1.0.1
Posted by Jason Smith <jh...@iriscouch.com>.
I once made a very simple CouchDB undelete tool. It scans your disk
device for anything that looks like the on-disk CouchDB JSON format.
https://github.com/jhs/grep_couch
I've recovered data with it, but notably, the _id and _rev are *not*
stored with the rest of a document, so you tend to get lots of docs
with no _id field. (I'm considering always having an "id" field to
dupe the "_id" in case I ever have to do that again.)
On Mon, Jun 6, 2011 at 3:35 PM, René Brüntrup <br...@imfl.de> wrote:
> Hello!
>
>> 1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
>
> We could not find the missing data in any of the replicated files. Each
> backup has an increasing number of documents with the least amount of
> missing data in the newest backup. Access logs from the 2011-03-08 are
> unfortunately missing, because the log rotation already removed them.
>
>> 2) Is it possible that you've inadvertently restored the database file from a backup?
>
> No backup was created at this date and we do not have any mechanisms
> that could automatically restore the backups.
>
>> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
>
> Our system does not work without an initialized database that contains a
> number of user account and definition documents. But even if such an
> initialized database would have been available underneath the encrypted
> volume we would have noticed a data loss after changing the database,
> because the system was already in use before the 2011-03-08.
>
>
> We will check again, if there is a database file underneath the
> encrypted volume, but we cannot stop the system right now. When
> replicating the database we just noticed, that the timestamp of the
> source database was updated. The replication processes that where
> started after the 2011-03-08 and before the reboot of the server did not
> change the timestamp of the source database.
>
>
> Regards,
> René
>
>
>
--
Iris Couch
Re: Data loss in CouchDB 1.0.1
Posted by René Brüntrup <br...@imfl.de>.
Hello!
> 1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
We could not find the missing data in any of the replicated files. Each
backup has an increasing number of documents with the least amount of
missing data in the newest backup. Access logs from the 2011-03-08 are
unfortunately missing, because the log rotation already removed them.
> 2) Is it possible that you've inadvertently restored the database file from a backup?
No backup was created at this date and we do not have any mechanisms
that could automatically restore the backups.
> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
Our system does not work without an initialized database that contains a
number of user account and definition documents. But even if such an
initialized database would have been available underneath the encrypted
volume we would have noticed a data loss after changing the database,
because the system was already in use before the 2011-03-08.
We will check again, if there is a database file underneath the
encrypted volume, but we cannot stop the system right now. When
replicating the database we just noticed, that the timestamp of the
source database was updated. The replication processes that where
started after the 2011-03-08 and before the reboot of the server did not
change the timestamp of the source database.
Regards,
René
Re: Data loss in CouchDB 1.0.1
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Jun 3, 2011 at 9:54 AM, Adam Kocoloski <ko...@apache.org> wrote:
> On Jun 3, 2011, at 9:21 AM, René Brüntrup wrote:
>
>> Dear CouchDB Community,
>>
>> After restarting our productive Ubuntu 10.10 server running the official
>> Ubuntu CouchDB 1.0.1 package we observed a data loss in our database.
>>
>> Before restarting the server the application using the database worked
>> without any problems. After restarting the server some recent
>> modifications were missing from the system and we started investigating
>> the problems:
>> The last change time of the CouchDB database file dated back about two
>> months. A hex dump of the CouchDB file showed no changes after this
>> point of time. Regular database backups that where created by
>> replicating into another database contained more recent data, just the
>> main database seemed to have stopped recording changes to disc after
>> 2011-08-03. All database files are stored on the same encrypted LVM
>> volume.
>>
>> The problem sounds a lot like writer process bug in 1.0.0: All data
>> seems to be kept in RAM, but never written to disc. However, we verified
>> that Ubuntu uses the patched code and the conditions that could cause
>> the problems in version 1.0.0 do not even occur in our system.
>>
>> So far we were not able to reproduce the behavior.
>>
>> Thank you for reading and we appreciate any help.
>>
>> Kind Regards
>> René Brüntrup
>
> Hi René, thanks for this report. Can you clarify a couple of things for us:
>
> 1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
>
> 2) Is it possible that you've inadvertently restored the database file from a backup?
>
> 3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
>
Just a quick clarification as I had to have this explained on IRC.
Writing "underneath" the LVM volume means opening a file in a path
under the mount point before mounting the encrypted volume. The
assumption being that the LVM mount happens late in the boot sequence
so it might be possible that it was mounted after CouchDB had already
been started and opened file handles to the underlying filesystem.
> CouchDB only allows clients -- including the replicator -- to read documents that have been written to disk (though not necessarily synced). The bug in 1.0.0 concerned the writing of database headers. The database file itself was definitely being updated.
>
> We take reports of this nature quite seriously and will be watching this thread closely. Regards,
>
> --
> Adam Kocoloski
>
>
>
Re: Data loss in CouchDB 1.0.1
Posted by Adam Kocoloski <ko...@apache.org>.
On Jun 3, 2011, at 9:21 AM, René Brüntrup wrote:
> Dear CouchDB Community,
>
> After restarting our productive Ubuntu 10.10 server running the official
> Ubuntu CouchDB 1.0.1 package we observed a data loss in our database.
>
> Before restarting the server the application using the database worked
> without any problems. After restarting the server some recent
> modifications were missing from the system and we started investigating
> the problems:
> The last change time of the CouchDB database file dated back about two
> months. A hex dump of the CouchDB file showed no changes after this
> point of time. Regular database backups that where created by
> replicating into another database contained more recent data, just the
> main database seemed to have stopped recording changes to disc after
> 2011-08-03. All database files are stored on the same encrypted LVM
> volume.
>
> The problem sounds a lot like writer process bug in 1.0.0: All data
> seems to be kept in RAM, but never written to disc. However, we verified
> that Ubuntu uses the patched code and the conditions that could cause
> the problems in version 1.0.0 do not even occur in our system.
>
> So far we were not able to reproduce the behavior.
>
> Thank you for reading and we appreciate any help.
>
> Kind Regards
> René Brüntrup
Hi René, thanks for this report. Can you clarify a couple of things for us:
1) Are you certain that you were in fact writing to the database on this server and not the replica? Can you share some access logs towards that end?
2) Is it possible that you've inadvertently restored the database file from a backup?
3) Is it possible that you were writing "underneath" the encrypted LVM volume for the past two months?
CouchDB only allows clients -- including the replicator -- to read documents that have been written to disk (though not necessarily synced). The bug in 1.0.0 concerned the writing of database headers. The database file itself was definitely being updated.
We take reports of this nature quite seriously and will be watching this thread closely. Regards,
--
Adam Kocoloski