You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Jan Lehnardt <ja...@apache.org> on 2010/08/09 18:09:24 UTC
data recovery tool progress
Hi All,
Filipe jumped in to start working on the recovery tool, but he isn't done yet.
Here's the current patch:
http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
it is not done and very early, but any help on this is greatly appreciated.
The current state is (in Filipe's words):
- i can detect that a file needs repair
- and get the last btree roots from it
- "only" missing: get last db seq num
- write new header
- and deal with the local docs btree (if exists)
Thanks!
Jan
--
Re: data recovery tool progress
Posted by Filipe David Manana <fd...@apache.org>.
On Wed, Aug 11, 2010 at 3:52 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
> had packaged the whole thing up as an escript + some .beams. If we can get
> it down to a single file a la rebar that would be a pretty sweet way to
> deliver the repair tool in my opinion.
>
+1
>
> Adam
>
> On Aug 10, 2010, at 10:40 PM, Mikeal Rogers wrote:
>
> > Ok, latest code has been tested against every db that I have and it works
> > great.
> >
> > What are our next steps here?
> >
> > I'd like to get this out to all the people who didn't feel comfortable
> send
> > me their db to test against before we release it more widely.
> >
> > -Mikeal
> >
> > On Tue, Aug 10, 2010 at 6:11 PM, Mikeal Rogers <mikeal.rogers@gmail.com
> >wrote:
> >
> >> Found one issue, we weren't picking up design docs because it didn't
> have
> >> admin privileges.
> >>
> >> Adam fixed it and pushed and I've verified that it works now.
> >>
> >> I wrote a little node script to show all recovered documents and expose
> any
> >> documents that didn't make it in to lost+found.
> >>
> >> http://github.com/mikeal/couchtest/blob/master/validate.js
> >>
> >> Requires request, `npm install request`.
> >>
> >> I'm now running recover on all the test db's I have and running the
> >> validation script against them.
> >>
> >> -Mikeal
> >>
> >>
> >> On Tue, Aug 10, 2010 at 1:34 PM, Mikeal Rogers <mikeal.rogers@gmail.com
> >wrote:
> >>
> >>> I have some timing number for the new code.
> >>>
> >>> multi_conflict has 200 lost documents and 201 documents total after
> >>> recovery.
> >>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
> >>> {25217069,ok}
> >>> 25 seconds
> >>>
> >>> Something funky is going on here. Investigating.
> >>> 1> timer:tc(couch_db_repair, make_lost_and_found,
> >>> ["multi_conflict_with_attach"]).
> >>> {654782,ok}
> >>> .6 seconds
> >>>
> >>> This db has 124969 documents in it.
> >>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["testwritesdb"]).
> >>> {1381969304,ok}
> >>> 23 minutes
> >>>
> >>> This database is about 500megs and 46660 before recovery and 46801
> after.
> >>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["prod"]).
> >>> {2329669113,ok}
> >>> 38.8 minutes
> >>>
> >>> -Mikeal
> >>>
> >>> On Tue, Aug 10, 2010 at 12:06 PM, Adam Kocoloski <kocolosk@apache.org
> >wrote:
> >>>
> >>>> Good idea. Now we've got
> >>>>
> >>>>> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
> >>>> bytes at 1380102
> >>>>> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
> >>>> bytes at 331526
> >>>>> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 331526
> >>>> bytes at 0
> >>>>> [info] [<0.33.0>] couch_db_repair writing 12 updates to
> >>>> lost+found/testwritesdb
> >>>>> [info] [<0.33.0>] couch_db_repair writing 9 updates to
> >>>> lost+found/testwritesdb
> >>>>> [info] [<0.33.0>] couch_db_repair writing 8 updates to
> >>>> lost+found/testwritesdb
> >>>>
> >>>> Adam
> >>>>
> >>>> On Aug 10, 2010, at 2:29 PM, Robert Newson wrote:
> >>>>
> >>>>> It took 20 minutes before the first 'update' line came out, but now
> >>>>> seems to be recovering smoothly. machine load is back down to sane
> >>>>> levels.
> >>>>>
> >>>>> Suggest feedback during the hunting phase.
> >>>>>
> >>>>> B.
> >>>>>
> >>>>> On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <kocolosk@apache.org
> >
> >>>> wrote:
> >>>>>> Thanks for the crosscheck. I'm not aware of anything in the node
> >>>> finder that would cause it to struggle mightily with healthy DBs. It
> pretty
> >>>> much ignores the health of the DB, in fact. Would be interested to
> hear
> >>>> more.
> >>>>>>
> >>>>>> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
> >>>>>>
> >>>>>>> I verified the new code's ability to repair the testwritesdb.
> system
> >>>>>>> load was smooth from start to finish.
> >>>>>>>
> >>>>>>> I started a further test on a different (healthy) database and
> system
> >>>>>>> load was severe again, just collecting the roots (the lost+found db
> >>>>>>> was not yet created when I aborted the attempt). I suspect the fact
> >>>>>>> that it's healthy is the issue, so if I'm right, perhaps a warning
> is
> >>>>>>> useful.
> >>>>>>>
> >>>>>>> B.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <
> kocolosk@apache.org>
> >>>> wrote:
> >>>>>>>> Another update. This morning I took a different tack and, rather
> >>>> than try to find root nodes, I just looked for all kv_nodes in the
> file and
> >>>> treated each of those as a separate virtual DB to be replicated. This
> >>>> reduces the algorithmic complexity of the repair, and it looks like
> >>>> testwritesdb repairs in ~30 minutes or so. Also, this method results
> in the
> >>>> lost+found DB containing every document, not just the missing ones.
> >>>>>>>>
> >>>>>>>> My branch does not currently include Randall's parallelization of
> >>>> the replications. It's still CPU-limited, so that may be a worthwhile
> >>>> optimization. On the other hand, I think we may be reaching a stage
> at
> >>>> which performance for this repair tool is 'good enough', and pmaps can
> make
> >>>> error handling a bit dicey.
> >>>>>>>>
> >>>>>>>> In short, I think this tool is now in good shape.
> >>>>>>>>
> >>>>>>>> http://github.com/kocolosk/couchdb/tree/db_repair
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
Also, an abrupt shutdown with delayed_commits=true might orphan some
data anyway, so the warning on startup might encourage more people to
disable it.
On Fri, Aug 13, 2010 at 3:49 PM, Robert Newson <ro...@gmail.com> wrote:
> A user (#herman) on IRC today reported slow startups for couchdb. I
> speculated that he'd hit the data loss bug and that couchdb was
> scanning backwards for a header. This turned out to be the case.
> Interestingly this was verified with a strace call, watching the read
> calls use earlier and earlier offsets.
>
> Should we consider a tweak to the tool, or couchdb itself, to report a
> warning if we have to seek back very far to find a header? Obviously
> it would a heuristic but there would be no real downside to the odd
> false positive since the recovery tool and subsequent replication will
> amount to a no-op.
>
> fyi: Reading the couchdb database (90G) with 'dd' took 22 minutes, but
> couchdb's backward scanning took 3 hours.
>
> B.
>
> On Fri, Aug 13, 2010 at 3:05 PM, J Chris Anderson <jc...@apache.org> wrote:
>>
>> On Aug 12, 2010, at 11:38 PM, Mikeal Rogers wrote:
>>
>>> I tested the latest code in recover-couchdb and it looks great.
>>
>> We need to package this so that it is useable by end-users, and put a link to it on http://couchdb.apache.org/notice/1.0.1.html
>>
>> I'm the last guy who knows what that would mean... anyone? I think we should do this today.
>>
>> Do we need to do anything formal and time consuming before linking to the recovery tool / process from that page?
>>
>> Also, someone needs to write up the how-to instructions, along with a description of what to expect.
>>
>> Chris
>>
>>>
>>> -Mikeal
>>>
>>> On Thu, Aug 12, 2010 at 2:33 PM, J Chris Anderson <jc...@apache.org> wrote:
>>>
>>>>
>>>> On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>>>>
>>>>>
>>>>> On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
>>>>>
>>>>>> Right, and jchris' db_repair branch includes my patches for DB reader
>>>> _admin access and a more useful progress report in the replication phase of
>>>> the repair.
>>>>>>
>>>>>
>>>>> I've updated the repair branch with everyone's code. I think it is
>>>> faster, due to Adam's idea that if we run the merges in reverse order, those
>>>> near the front of the file are more likely to be no-ops, so less work is
>>>> done over all.
>>>>>
>>>>> Mikeal will be testing for correctness. Could other's please use it and
>>>> test for usability as well. Latest code (with instructions) is here:
>>>>>
>>>>> http://github.com/jhs/recover-couchdb/
>>>>>
>>>>> Which points at http://github.com/jchris/couchdb/tree/db_repair for the
>>>> repair code.
>>>>>
>>>>> One thing I am not clear about (need better docs) is, do we need to
>>>> replicate the original db to the lost+found db (or vice-versa), after
>>>> recovery is complete?
>>>>>
>>>>
>>>> Also, we should be clear about what the semantics for this are. It can
>>>> potentially introduce conflicts if some writes were repeated after restarts.
>>>> Should it always be a noop on dbs that are clean w/r/t the bug?
>>>>
>>>> Chris
>>>>
>>>>> Chris
>>>>>
>>>>>> Adam
>>>>>>
>>>>>> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>>>>>>
>>>>>>> The code is updated with the following changes:
>>>>>>> 1. Adhere to the lost+found/databasename custom...
>>>>>>> 2. ...except databases starting with _, which goes into
>>>>>>> _system/databasename
>>>>>>> 3. Sync up with jchris's db_repair branch
>>>>>>>
>>>>>>> (About #2, I started with _/database but I think it's too easy to miss
>>>> at
>>>>>>> the command line.)
>>>>>>>
>>>>>>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> A few bug reports from my testing:
>>>>>>>>
>>>>>>>> I launched with this command, as specified in the README:
>>>>>>>>
>>>>>>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec
>>>> ./recover_couchdb
>>>>>>>> {} \;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> First of all, it chokes on my _users and _replicator db:
>>>>>>>>
>>>>>>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at
>>>> 0
>>>>>>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>>>>>>> {error,illegal_database_name}}
>>>>>>>>
>>>>>>>> That second [error] line is repeated many many times (once per merge I
>>>>>>>> think). I think the issue is that _users is hard-coded to be OK, but
>>>>>>>> _users_lost+found is not. So we should do something about that, maybe
>>>> if a
>>>>>>>> db-name starts with _ we should call the lost and found
>>>> a_users_lost+found
>>>>>>>> (_ sorts at the top, so "a" will be near it and legal).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When a database has readers defined in the security object, the tool
>>>> is
>>>>>>>> unable to open them (the reading part of the repair tool needs to have
>>>> the
>>>>>>>> _admin userCtx, not just the writer).
>>>>>>>>
>>>>>>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined}
>>>> vs
>>>>>>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>>>>>>> escript: exception throw: {unauthorized,<<"You are not authorized to
>>>> access
>>>>>>>> this db.">>}
>>>>>>>> in function couch_db:open/2
>>>>>>>> in call from couch_db_repair:make_lost_and_found/3
>>>>>>>> in call from recover_couchdb:main/1
>>>>>>>> in call from escript:run/2
>>>>>>>> in call from escript:start/1
>>>>>>>> in call from init:start_it/1
>>>>>>>> in call from init:start_em/1
>>>>>>>>
>>>>>>>>
>>>>>>>> It would also be helpful if the status lines could say something more
>>>> than
>>>>>>>>
>>>>>>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to
>>>> bench_lost+found
>>>>>>>>
>>>>>>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>>>>>>
>>>>>>>>
>>>>>>>> I will patch the first few, I'd love help from someone on the last
>>>> one.
>>>>>>>> I'll be on IRC.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>>>>>>
>>>>>>>>>> Hi, Jason.
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <kocolosk@apache.org
>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>>>>>>> that he
>>>>>>>>>>>> had packaged the whole thing up as an escript + some .beams. If
>>>> we
>>>>>>>> can get
>>>>>>>>>>>> it down to a single file a la rebar that would be a pretty sweet
>>>> way
>>>>>>>> to
>>>>>>>>>>>> deliver the repair tool in my opinion.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think it is important that we package and release this, if it is
>>>> ready.
>>>>>>>> We should link to it from the bug description page, the project home
>>>> page,
>>>>>>>> as well as blog about it, etc. What is the point of working feverishly
>>>> on a
>>>>>>>> recovery tool if we don't go the last mile?
>>>>>>>>>
>>>>>>>>> I am testing it now on my database directory to make sure it doesn't
>>>> harm
>>>>>>>> anything (I was never subject to the bug, which is probably where most
>>>>>>>> people are, but they might run it anyway.)
>>>>>>>>>
>>>>>>>>> As it stands the submodules thing can't be part of the release, we
>>>> need
>>>>>>>> to package it up as a single zip file or something.
>>>>>>>>>
>>>>>>>>> Is there anything else that needs to be done before we can release
>>>> this?
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jason Smith
>>>>>>>>>> Couchio Hosting
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jason Smith
>>>>>>> Couchio Hosting
>>>>>>
>>>>>
>>>>
>>>>
>>
>>
>
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
A user (#herman) on IRC today reported slow startups for couchdb. I
speculated that he'd hit the data loss bug and that couchdb was
scanning backwards for a header. This turned out to be the case.
Interestingly this was verified with a strace call, watching the read
calls use earlier and earlier offsets.
Should we consider a tweak to the tool, or couchdb itself, to report a
warning if we have to seek back very far to find a header? Obviously
it would a heuristic but there would be no real downside to the odd
false positive since the recovery tool and subsequent replication will
amount to a no-op.
fyi: Reading the couchdb database (90G) with 'dd' took 22 minutes, but
couchdb's backward scanning took 3 hours.
B.
On Fri, Aug 13, 2010 at 3:05 PM, J Chris Anderson <jc...@apache.org> wrote:
>
> On Aug 12, 2010, at 11:38 PM, Mikeal Rogers wrote:
>
>> I tested the latest code in recover-couchdb and it looks great.
>
> We need to package this so that it is useable by end-users, and put a link to it on http://couchdb.apache.org/notice/1.0.1.html
>
> I'm the last guy who knows what that would mean... anyone? I think we should do this today.
>
> Do we need to do anything formal and time consuming before linking to the recovery tool / process from that page?
>
> Also, someone needs to write up the how-to instructions, along with a description of what to expect.
>
> Chris
>
>>
>> -Mikeal
>>
>> On Thu, Aug 12, 2010 at 2:33 PM, J Chris Anderson <jc...@apache.org> wrote:
>>
>>>
>>> On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>>>
>>>>
>>>> On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
>>>>
>>>>> Right, and jchris' db_repair branch includes my patches for DB reader
>>> _admin access and a more useful progress report in the replication phase of
>>> the repair.
>>>>>
>>>>
>>>> I've updated the repair branch with everyone's code. I think it is
>>> faster, due to Adam's idea that if we run the merges in reverse order, those
>>> near the front of the file are more likely to be no-ops, so less work is
>>> done over all.
>>>>
>>>> Mikeal will be testing for correctness. Could other's please use it and
>>> test for usability as well. Latest code (with instructions) is here:
>>>>
>>>> http://github.com/jhs/recover-couchdb/
>>>>
>>>> Which points at http://github.com/jchris/couchdb/tree/db_repair for the
>>> repair code.
>>>>
>>>> One thing I am not clear about (need better docs) is, do we need to
>>> replicate the original db to the lost+found db (or vice-versa), after
>>> recovery is complete?
>>>>
>>>
>>> Also, we should be clear about what the semantics for this are. It can
>>> potentially introduce conflicts if some writes were repeated after restarts.
>>> Should it always be a noop on dbs that are clean w/r/t the bug?
>>>
>>> Chris
>>>
>>>> Chris
>>>>
>>>>> Adam
>>>>>
>>>>> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>>>>>
>>>>>> The code is updated with the following changes:
>>>>>> 1. Adhere to the lost+found/databasename custom...
>>>>>> 2. ...except databases starting with _, which goes into
>>>>>> _system/databasename
>>>>>> 3. Sync up with jchris's db_repair branch
>>>>>>
>>>>>> (About #2, I started with _/database but I think it's too easy to miss
>>> at
>>>>>> the command line.)
>>>>>>
>>>>>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> A few bug reports from my testing:
>>>>>>>
>>>>>>> I launched with this command, as specified in the README:
>>>>>>>
>>>>>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec
>>> ./recover_couchdb
>>>>>>> {} \;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> First of all, it chokes on my _users and _replicator db:
>>>>>>>
>>>>>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at
>>> 0
>>>>>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>>>>>> {error,illegal_database_name}}
>>>>>>>
>>>>>>> That second [error] line is repeated many many times (once per merge I
>>>>>>> think). I think the issue is that _users is hard-coded to be OK, but
>>>>>>> _users_lost+found is not. So we should do something about that, maybe
>>> if a
>>>>>>> db-name starts with _ we should call the lost and found
>>> a_users_lost+found
>>>>>>> (_ sorts at the top, so "a" will be near it and legal).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> When a database has readers defined in the security object, the tool
>>> is
>>>>>>> unable to open them (the reading part of the repair tool needs to have
>>> the
>>>>>>> _admin userCtx, not just the writer).
>>>>>>>
>>>>>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined}
>>> vs
>>>>>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>>>>>> escript: exception throw: {unauthorized,<<"You are not authorized to
>>> access
>>>>>>> this db.">>}
>>>>>>> in function couch_db:open/2
>>>>>>> in call from couch_db_repair:make_lost_and_found/3
>>>>>>> in call from recover_couchdb:main/1
>>>>>>> in call from escript:run/2
>>>>>>> in call from escript:start/1
>>>>>>> in call from init:start_it/1
>>>>>>> in call from init:start_em/1
>>>>>>>
>>>>>>>
>>>>>>> It would also be helpful if the status lines could say something more
>>> than
>>>>>>>
>>>>>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to
>>> bench_lost+found
>>>>>>>
>>>>>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>>>>>
>>>>>>>
>>>>>>> I will patch the first few, I'd love help from someone on the last
>>> one.
>>>>>>> I'll be on IRC.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>>>>>
>>>>>>>>> Hi, Jason.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <kocolosk@apache.org
>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>>>>>> that he
>>>>>>>>>>> had packaged the whole thing up as an escript + some .beams. If
>>> we
>>>>>>> can get
>>>>>>>>>>> it down to a single file a la rebar that would be a pretty sweet
>>> way
>>>>>>> to
>>>>>>>>>>> deliver the repair tool in my opinion.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think it is important that we package and release this, if it is
>>> ready.
>>>>>>> We should link to it from the bug description page, the project home
>>> page,
>>>>>>> as well as blog about it, etc. What is the point of working feverishly
>>> on a
>>>>>>> recovery tool if we don't go the last mile?
>>>>>>>>
>>>>>>>> I am testing it now on my database directory to make sure it doesn't
>>> harm
>>>>>>> anything (I was never subject to the bug, which is probably where most
>>>>>>> people are, but they might run it anyway.)
>>>>>>>>
>>>>>>>> As it stands the submodules thing can't be part of the release, we
>>> need
>>>>>>> to package it up as a single zip file or something.
>>>>>>>>
>>>>>>>> Is there anything else that needs to be done before we can release
>>> this?
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jason Smith
>>>>>>>>> Couchio Hosting
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jason Smith
>>>>>> Couchio Hosting
>>>>>
>>>>
>>>
>>>
>
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 13, 2010, at 9:26 AM, J Chris Anderson wrote:
> Here is my first pass at notes on the repair tool.
>
> I'd like to get this on the Apache website today so we can publicize it:
>
> http://wiki.couchone.com/page/repair-tool
>
Sitting on our hands wasn't doing anyone any good, so I tweeted this as CouchDB.
http://twitter.com/CouchDB/status/21083958040
I still think we should put this on the Apache site and give it a proper announcement.
Chris
> Please read, test, edit, give feedback, etc.
>
> Thanks,
> Chris
>
> On Aug 13, 2010, at 7:05 AM, J Chris Anderson wrote:
>
>>
>> On Aug 12, 2010, at 11:38 PM, Mikeal Rogers wrote:
>>
>>> I tested the latest code in recover-couchdb and it looks great.
>>
>> We need to package this so that it is useable by end-users, and put a link to it on http://couchdb.apache.org/notice/1.0.1.html
>>
>> I'm the last guy who knows what that would mean... anyone? I think we should do this today.
>>
>> Do we need to do anything formal and time consuming before linking to the recovery tool / process from that page?
>>
>> Also, someone needs to write up the how-to instructions, along with a description of what to expect.
>>
>> Chris
>>
>>>
>>> -Mikeal
>>>
>>> On Thu, Aug 12, 2010 at 2:33 PM, J Chris Anderson <jc...@apache.org> wrote:
>>>
>>>>
>>>> On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>>>>
>>>>>
>>>>> On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
>>>>>
>>>>>> Right, and jchris' db_repair branch includes my patches for DB reader
>>>> _admin access and a more useful progress report in the replication phase of
>>>> the repair.
>>>>>>
>>>>>
>>>>> I've updated the repair branch with everyone's code. I think it is
>>>> faster, due to Adam's idea that if we run the merges in reverse order, those
>>>> near the front of the file are more likely to be no-ops, so less work is
>>>> done over all.
>>>>>
>>>>> Mikeal will be testing for correctness. Could other's please use it and
>>>> test for usability as well. Latest code (with instructions) is here:
>>>>>
>>>>> http://github.com/jhs/recover-couchdb/
>>>>>
>>>>> Which points at http://github.com/jchris/couchdb/tree/db_repair for the
>>>> repair code.
>>>>>
>>>>> One thing I am not clear about (need better docs) is, do we need to
>>>> replicate the original db to the lost+found db (or vice-versa), after
>>>> recovery is complete?
>>>>>
>>>>
>>>> Also, we should be clear about what the semantics for this are. It can
>>>> potentially introduce conflicts if some writes were repeated after restarts.
>>>> Should it always be a noop on dbs that are clean w/r/t the bug?
>>>>
>>>> Chris
>>>>
>>>>> Chris
>>>>>
>>>>>> Adam
>>>>>>
>>>>>> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>>>>>>
>>>>>>> The code is updated with the following changes:
>>>>>>> 1. Adhere to the lost+found/databasename custom...
>>>>>>> 2. ...except databases starting with _, which goes into
>>>>>>> _system/databasename
>>>>>>> 3. Sync up with jchris's db_repair branch
>>>>>>>
>>>>>>> (About #2, I started with _/database but I think it's too easy to miss
>>>> at
>>>>>>> the command line.)
>>>>>>>
>>>>>>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> A few bug reports from my testing:
>>>>>>>>
>>>>>>>> I launched with this command, as specified in the README:
>>>>>>>>
>>>>>>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec
>>>> ./recover_couchdb
>>>>>>>> {} \;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> First of all, it chokes on my _users and _replicator db:
>>>>>>>>
>>>>>>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at
>>>> 0
>>>>>>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>>>>>>> {error,illegal_database_name}}
>>>>>>>>
>>>>>>>> That second [error] line is repeated many many times (once per merge I
>>>>>>>> think). I think the issue is that _users is hard-coded to be OK, but
>>>>>>>> _users_lost+found is not. So we should do something about that, maybe
>>>> if a
>>>>>>>> db-name starts with _ we should call the lost and found
>>>> a_users_lost+found
>>>>>>>> (_ sorts at the top, so "a" will be near it and legal).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When a database has readers defined in the security object, the tool
>>>> is
>>>>>>>> unable to open them (the reading part of the repair tool needs to have
>>>> the
>>>>>>>> _admin userCtx, not just the writer).
>>>>>>>>
>>>>>>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined}
>>>> vs
>>>>>>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>>>>>>> escript: exception throw: {unauthorized,<<"You are not authorized to
>>>> access
>>>>>>>> this db.">>}
>>>>>>>> in function couch_db:open/2
>>>>>>>> in call from couch_db_repair:make_lost_and_found/3
>>>>>>>> in call from recover_couchdb:main/1
>>>>>>>> in call from escript:run/2
>>>>>>>> in call from escript:start/1
>>>>>>>> in call from init:start_it/1
>>>>>>>> in call from init:start_em/1
>>>>>>>>
>>>>>>>>
>>>>>>>> It would also be helpful if the status lines could say something more
>>>> than
>>>>>>>>
>>>>>>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to
>>>> bench_lost+found
>>>>>>>>
>>>>>>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>>>>>>
>>>>>>>>
>>>>>>>> I will patch the first few, I'd love help from someone on the last
>>>> one.
>>>>>>>> I'll be on IRC.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>>>>>>
>>>>>>>>>> Hi, Jason.
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <kocolosk@apache.org
>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>>>>>>> that he
>>>>>>>>>>>> had packaged the whole thing up as an escript + some .beams. If
>>>> we
>>>>>>>> can get
>>>>>>>>>>>> it down to a single file a la rebar that would be a pretty sweet
>>>> way
>>>>>>>> to
>>>>>>>>>>>> deliver the repair tool in my opinion.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think it is important that we package and release this, if it is
>>>> ready.
>>>>>>>> We should link to it from the bug description page, the project home
>>>> page,
>>>>>>>> as well as blog about it, etc. What is the point of working feverishly
>>>> on a
>>>>>>>> recovery tool if we don't go the last mile?
>>>>>>>>>
>>>>>>>>> I am testing it now on my database directory to make sure it doesn't
>>>> harm
>>>>>>>> anything (I was never subject to the bug, which is probably where most
>>>>>>>> people are, but they might run it anyway.)
>>>>>>>>>
>>>>>>>>> As it stands the submodules thing can't be part of the release, we
>>>> need
>>>>>>>> to package it up as a single zip file or something.
>>>>>>>>>
>>>>>>>>> Is there anything else that needs to be done before we can release
>>>> this?
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jason Smith
>>>>>>>>>> Couchio Hosting
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jason Smith
>>>>>>> Couchio Hosting
>>>>>>
>>>>>
>>>>
>>>>
>>
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
Here is my first pass at notes on the repair tool.
I'd like to get this on the Apache website today so we can publicize it:
http://wiki.couchone.com/page/repair-tool
Please read, test, edit, give feedback, etc.
Thanks,
Chris
On Aug 13, 2010, at 7:05 AM, J Chris Anderson wrote:
>
> On Aug 12, 2010, at 11:38 PM, Mikeal Rogers wrote:
>
>> I tested the latest code in recover-couchdb and it looks great.
>
> We need to package this so that it is useable by end-users, and put a link to it on http://couchdb.apache.org/notice/1.0.1.html
>
> I'm the last guy who knows what that would mean... anyone? I think we should do this today.
>
> Do we need to do anything formal and time consuming before linking to the recovery tool / process from that page?
>
> Also, someone needs to write up the how-to instructions, along with a description of what to expect.
>
> Chris
>
>>
>> -Mikeal
>>
>> On Thu, Aug 12, 2010 at 2:33 PM, J Chris Anderson <jc...@apache.org> wrote:
>>
>>>
>>> On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>>>
>>>>
>>>> On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
>>>>
>>>>> Right, and jchris' db_repair branch includes my patches for DB reader
>>> _admin access and a more useful progress report in the replication phase of
>>> the repair.
>>>>>
>>>>
>>>> I've updated the repair branch with everyone's code. I think it is
>>> faster, due to Adam's idea that if we run the merges in reverse order, those
>>> near the front of the file are more likely to be no-ops, so less work is
>>> done over all.
>>>>
>>>> Mikeal will be testing for correctness. Could other's please use it and
>>> test for usability as well. Latest code (with instructions) is here:
>>>>
>>>> http://github.com/jhs/recover-couchdb/
>>>>
>>>> Which points at http://github.com/jchris/couchdb/tree/db_repair for the
>>> repair code.
>>>>
>>>> One thing I am not clear about (need better docs) is, do we need to
>>> replicate the original db to the lost+found db (or vice-versa), after
>>> recovery is complete?
>>>>
>>>
>>> Also, we should be clear about what the semantics for this are. It can
>>> potentially introduce conflicts if some writes were repeated after restarts.
>>> Should it always be a noop on dbs that are clean w/r/t the bug?
>>>
>>> Chris
>>>
>>>> Chris
>>>>
>>>>> Adam
>>>>>
>>>>> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>>>>>
>>>>>> The code is updated with the following changes:
>>>>>> 1. Adhere to the lost+found/databasename custom...
>>>>>> 2. ...except databases starting with _, which goes into
>>>>>> _system/databasename
>>>>>> 3. Sync up with jchris's db_repair branch
>>>>>>
>>>>>> (About #2, I started with _/database but I think it's too easy to miss
>>> at
>>>>>> the command line.)
>>>>>>
>>>>>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> A few bug reports from my testing:
>>>>>>>
>>>>>>> I launched with this command, as specified in the README:
>>>>>>>
>>>>>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec
>>> ./recover_couchdb
>>>>>>> {} \;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> First of all, it chokes on my _users and _replicator db:
>>>>>>>
>>>>>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at
>>> 0
>>>>>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>>>>>> {error,illegal_database_name}}
>>>>>>>
>>>>>>> That second [error] line is repeated many many times (once per merge I
>>>>>>> think). I think the issue is that _users is hard-coded to be OK, but
>>>>>>> _users_lost+found is not. So we should do something about that, maybe
>>> if a
>>>>>>> db-name starts with _ we should call the lost and found
>>> a_users_lost+found
>>>>>>> (_ sorts at the top, so "a" will be near it and legal).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> When a database has readers defined in the security object, the tool
>>> is
>>>>>>> unable to open them (the reading part of the repair tool needs to have
>>> the
>>>>>>> _admin userCtx, not just the writer).
>>>>>>>
>>>>>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined}
>>> vs
>>>>>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>>>>>> escript: exception throw: {unauthorized,<<"You are not authorized to
>>> access
>>>>>>> this db.">>}
>>>>>>> in function couch_db:open/2
>>>>>>> in call from couch_db_repair:make_lost_and_found/3
>>>>>>> in call from recover_couchdb:main/1
>>>>>>> in call from escript:run/2
>>>>>>> in call from escript:start/1
>>>>>>> in call from init:start_it/1
>>>>>>> in call from init:start_em/1
>>>>>>>
>>>>>>>
>>>>>>> It would also be helpful if the status lines could say something more
>>> than
>>>>>>>
>>>>>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to
>>> bench_lost+found
>>>>>>>
>>>>>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>>>>>
>>>>>>>
>>>>>>> I will patch the first few, I'd love help from someone on the last
>>> one.
>>>>>>> I'll be on IRC.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>>>>>
>>>>>>>>> Hi, Jason.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <kocolosk@apache.org
>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>>>>>> that he
>>>>>>>>>>> had packaged the whole thing up as an escript + some .beams. If
>>> we
>>>>>>> can get
>>>>>>>>>>> it down to a single file a la rebar that would be a pretty sweet
>>> way
>>>>>>> to
>>>>>>>>>>> deliver the repair tool in my opinion.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think it is important that we package and release this, if it is
>>> ready.
>>>>>>> We should link to it from the bug description page, the project home
>>> page,
>>>>>>> as well as blog about it, etc. What is the point of working feverishly
>>> on a
>>>>>>> recovery tool if we don't go the last mile?
>>>>>>>>
>>>>>>>> I am testing it now on my database directory to make sure it doesn't
>>> harm
>>>>>>> anything (I was never subject to the bug, which is probably where most
>>>>>>> people are, but they might run it anyway.)
>>>>>>>>
>>>>>>>> As it stands the submodules thing can't be part of the release, we
>>> need
>>>>>>> to package it up as a single zip file or something.
>>>>>>>>
>>>>>>>> Is there anything else that needs to be done before we can release
>>> this?
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jason Smith
>>>>>>>>> Couchio Hosting
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jason Smith
>>>>>> Couchio Hosting
>>>>>
>>>>
>>>
>>>
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 12, 2010, at 11:38 PM, Mikeal Rogers wrote:
> I tested the latest code in recover-couchdb and it looks great.
We need to package this so that it is useable by end-users, and put a link to it on http://couchdb.apache.org/notice/1.0.1.html
I'm the last guy who knows what that would mean... anyone? I think we should do this today.
Do we need to do anything formal and time consuming before linking to the recovery tool / process from that page?
Also, someone needs to write up the how-to instructions, along with a description of what to expect.
Chris
>
> -Mikeal
>
> On Thu, Aug 12, 2010 at 2:33 PM, J Chris Anderson <jc...@apache.org> wrote:
>
>>
>> On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>>
>>>
>>> On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
>>>
>>>> Right, and jchris' db_repair branch includes my patches for DB reader
>> _admin access and a more useful progress report in the replication phase of
>> the repair.
>>>>
>>>
>>> I've updated the repair branch with everyone's code. I think it is
>> faster, due to Adam's idea that if we run the merges in reverse order, those
>> near the front of the file are more likely to be no-ops, so less work is
>> done over all.
>>>
>>> Mikeal will be testing for correctness. Could other's please use it and
>> test for usability as well. Latest code (with instructions) is here:
>>>
>>> http://github.com/jhs/recover-couchdb/
>>>
>>> Which points at http://github.com/jchris/couchdb/tree/db_repair for the
>> repair code.
>>>
>>> One thing I am not clear about (need better docs) is, do we need to
>> replicate the original db to the lost+found db (or vice-versa), after
>> recovery is complete?
>>>
>>
>> Also, we should be clear about what the semantics for this are. It can
>> potentially introduce conflicts if some writes were repeated after restarts.
>> Should it always be a noop on dbs that are clean w/r/t the bug?
>>
>> Chris
>>
>>> Chris
>>>
>>>> Adam
>>>>
>>>> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>>>>
>>>>> The code is updated with the following changes:
>>>>> 1. Adhere to the lost+found/databasename custom...
>>>>> 2. ...except databases starting with _, which goes into
>>>>> _system/databasename
>>>>> 3. Sync up with jchris's db_repair branch
>>>>>
>>>>> (About #2, I started with _/database but I think it's too easy to miss
>> at
>>>>> the command line.)
>>>>>
>>>>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com>
>> wrote:
>>>>>
>>>>>> A few bug reports from my testing:
>>>>>>
>>>>>> I launched with this command, as specified in the README:
>>>>>>
>>>>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec
>> ./recover_couchdb
>>>>>> {} \;
>>>>>>
>>>>>>
>>>>>>
>>>>>> First of all, it chokes on my _users and _replicator db:
>>>>>>
>>>>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at
>> 0
>>>>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>>>>> {error,illegal_database_name}}
>>>>>>
>>>>>> That second [error] line is repeated many many times (once per merge I
>>>>>> think). I think the issue is that _users is hard-coded to be OK, but
>>>>>> _users_lost+found is not. So we should do something about that, maybe
>> if a
>>>>>> db-name starts with _ we should call the lost and found
>> a_users_lost+found
>>>>>> (_ sorts at the top, so "a" will be near it and legal).
>>>>>>
>>>>>>
>>>>>>
>>>>>> When a database has readers defined in the security object, the tool
>> is
>>>>>> unable to open them (the reading part of the repair tool needs to have
>> the
>>>>>> _admin userCtx, not just the writer).
>>>>>>
>>>>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined}
>> vs
>>>>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>>>>> escript: exception throw: {unauthorized,<<"You are not authorized to
>> access
>>>>>> this db.">>}
>>>>>> in function couch_db:open/2
>>>>>> in call from couch_db_repair:make_lost_and_found/3
>>>>>> in call from recover_couchdb:main/1
>>>>>> in call from escript:run/2
>>>>>> in call from escript:start/1
>>>>>> in call from init:start_it/1
>>>>>> in call from init:start_em/1
>>>>>>
>>>>>>
>>>>>> It would also be helpful if the status lines could say something more
>> than
>>>>>>
>>>>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to
>> bench_lost+found
>>>>>>
>>>>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>>>>
>>>>>>
>>>>>> I will patch the first few, I'd love help from someone on the last
>> one.
>>>>>> I'll be on IRC.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>>>>
>>>>>>>
>>>>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>>>>
>>>>>>>> Hi, Jason.
>>>>>>>>
>>>>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <kocolosk@apache.org
>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>>>>> that he
>>>>>>>>>> had packaged the whole thing up as an escript + some .beams. If
>> we
>>>>>> can get
>>>>>>>>>> it down to a single file a la rebar that would be a pretty sweet
>> way
>>>>>> to
>>>>>>>>>> deliver the repair tool in my opinion.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>>>>
>>>>>>>
>>>>>>> I think it is important that we package and release this, if it is
>> ready.
>>>>>> We should link to it from the bug description page, the project home
>> page,
>>>>>> as well as blog about it, etc. What is the point of working feverishly
>> on a
>>>>>> recovery tool if we don't go the last mile?
>>>>>>>
>>>>>>> I am testing it now on my database directory to make sure it doesn't
>> harm
>>>>>> anything (I was never subject to the bug, which is probably where most
>>>>>> people are, but they might run it anyway.)
>>>>>>>
>>>>>>> As it stands the submodules thing can't be part of the release, we
>> need
>>>>>> to package it up as a single zip file or something.
>>>>>>>
>>>>>>> Is there anything else that needs to be done before we can release
>> this?
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>>> --
>>>>>>>> Jason Smith
>>>>>>>> Couchio Hosting
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jason Smith
>>>>> Couchio Hosting
>>>>
>>>
>>
>>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
I tested the latest code in recover-couchdb and it looks great.
-Mikeal
On Thu, Aug 12, 2010 at 2:33 PM, J Chris Anderson <jc...@apache.org> wrote:
>
> On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>
> >
> > On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
> >
> >> Right, and jchris' db_repair branch includes my patches for DB reader
> _admin access and a more useful progress report in the replication phase of
> the repair.
> >>
> >
> > I've updated the repair branch with everyone's code. I think it is
> faster, due to Adam's idea that if we run the merges in reverse order, those
> near the front of the file are more likely to be no-ops, so less work is
> done over all.
> >
> > Mikeal will be testing for correctness. Could other's please use it and
> test for usability as well. Latest code (with instructions) is here:
> >
> > http://github.com/jhs/recover-couchdb/
> >
> > Which points at http://github.com/jchris/couchdb/tree/db_repair for the
> repair code.
> >
> > One thing I am not clear about (need better docs) is, do we need to
> replicate the original db to the lost+found db (or vice-versa), after
> recovery is complete?
> >
>
> Also, we should be clear about what the semantics for this are. It can
> potentially introduce conflicts if some writes were repeated after restarts.
> Should it always be a noop on dbs that are clean w/r/t the bug?
>
> Chris
>
> > Chris
> >
> >> Adam
> >>
> >> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
> >>
> >>> The code is updated with the following changes:
> >>> 1. Adhere to the lost+found/databasename custom...
> >>> 2. ...except databases starting with _, which goes into
> >>> _system/databasename
> >>> 3. Sync up with jchris's db_repair branch
> >>>
> >>> (About #2, I started with _/database but I think it's too easy to miss
> at
> >>> the command line.)
> >>>
> >>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com>
> wrote:
> >>>
> >>>> A few bug reports from my testing:
> >>>>
> >>>> I launched with this command, as specified in the README:
> >>>>
> >>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec
> ./recover_couchdb
> >>>> {} \;
> >>>>
> >>>>
> >>>>
> >>>> First of all, it chokes on my _users and _replicator db:
> >>>>
> >>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at
> 0
> >>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
> >>>> {error,illegal_database_name}}
> >>>>
> >>>> That second [error] line is repeated many many times (once per merge I
> >>>> think). I think the issue is that _users is hard-coded to be OK, but
> >>>> _users_lost+found is not. So we should do something about that, maybe
> if a
> >>>> db-name starts with _ we should call the lost and found
> a_users_lost+found
> >>>> (_ sorts at the top, so "a" will be near it and legal).
> >>>>
> >>>>
> >>>>
> >>>> When a database has readers defined in the security object, the tool
> is
> >>>> unable to open them (the reading part of the repair tool needs to have
> the
> >>>> _admin userCtx, not just the writer).
> >>>>
> >>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined}
> vs
> >>>> Names [<<"joe">>] Roles [<<"_admin">>]
> >>>> escript: exception throw: {unauthorized,<<"You are not authorized to
> access
> >>>> this db.">>}
> >>>> in function couch_db:open/2
> >>>> in call from couch_db_repair:make_lost_and_found/3
> >>>> in call from recover_couchdb:main/1
> >>>> in call from escript:run/2
> >>>> in call from escript:start/1
> >>>> in call from init:start_it/1
> >>>> in call from init:start_em/1
> >>>>
> >>>>
> >>>> It would also be helpful if the status lines could say something more
> than
> >>>>
> >>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to
> bench_lost+found
> >>>>
> >>>> Like maybe add a note like "about 23% complete" if at all possible.
> >>>>
> >>>>
> >>>> I will patch the first few, I'd love help from someone on the last
> one.
> >>>> I'll be on IRC.
> >>>>
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
> >>>>
> >>>>>
> >>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
> >>>>>
> >>>>>> Hi, Jason.
> >>>>>>
> >>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
> >>>>>>
> >>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <kocolosk@apache.org
> >
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
> >>>> that he
> >>>>>>>> had packaged the whole thing up as an escript + some .beams. If
> we
> >>>> can get
> >>>>>>>> it down to a single file a la rebar that would be a pretty sweet
> way
> >>>> to
> >>>>>>>> deliver the repair tool in my opinion.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Please check out http://github.com/jhs/repair-couchdb
> >>>>>>>
> >>>>>>
> >>>>>> I think you mean http://github.com/jhs/recover-couchdb
> >>>>>>
> >>>>>
> >>>>> I think it is important that we package and release this, if it is
> ready.
> >>>> We should link to it from the bug description page, the project home
> page,
> >>>> as well as blog about it, etc. What is the point of working feverishly
> on a
> >>>> recovery tool if we don't go the last mile?
> >>>>>
> >>>>> I am testing it now on my database directory to make sure it doesn't
> harm
> >>>> anything (I was never subject to the bug, which is probably where most
> >>>> people are, but they might run it anyway.)
> >>>>>
> >>>>> As it stands the submodules thing can't be part of the release, we
> need
> >>>> to package it up as a single zip file or something.
> >>>>>
> >>>>> Is there anything else that needs to be done before we can release
> this?
> >>>>>
> >>>>> Chris
> >>>>>
> >>>>>> --
> >>>>>> Jason Smith
> >>>>>> Couchio Hosting
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Jason Smith
> >>> Couchio Hosting
> >>
> >
>
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 12, 2010, at 2:15 PM, J Chris Anderson wrote:
>
> On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
>
>> Right, and jchris' db_repair branch includes my patches for DB reader _admin access and a more useful progress report in the replication phase of the repair.
>>
>
> I've updated the repair branch with everyone's code. I think it is faster, due to Adam's idea that if we run the merges in reverse order, those near the front of the file are more likely to be no-ops, so less work is done over all.
>
> Mikeal will be testing for correctness. Could other's please use it and test for usability as well. Latest code (with instructions) is here:
>
> http://github.com/jhs/recover-couchdb/
>
> Which points at http://github.com/jchris/couchdb/tree/db_repair for the repair code.
>
> One thing I am not clear about (need better docs) is, do we need to replicate the original db to the lost+found db (or vice-versa), after recovery is complete?
>
Also, we should be clear about what the semantics for this are. It can potentially introduce conflicts if some writes were repeated after restarts. Should it always be a noop on dbs that are clean w/r/t the bug?
Chris
> Chris
>
>> Adam
>>
>> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>>
>>> The code is updated with the following changes:
>>> 1. Adhere to the lost+found/databasename custom...
>>> 2. ...except databases starting with _, which goes into
>>> _system/databasename
>>> 3. Sync up with jchris's db_repair branch
>>>
>>> (About #2, I started with _/database but I think it's too easy to miss at
>>> the command line.)
>>>
>>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com> wrote:
>>>
>>>> A few bug reports from my testing:
>>>>
>>>> I launched with this command, as specified in the README:
>>>>
>>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec ./recover_couchdb
>>>> {} \;
>>>>
>>>>
>>>>
>>>> First of all, it chokes on my _users and _replicator db:
>>>>
>>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at 0
>>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>>> {error,illegal_database_name}}
>>>>
>>>> That second [error] line is repeated many many times (once per merge I
>>>> think). I think the issue is that _users is hard-coded to be OK, but
>>>> _users_lost+found is not. So we should do something about that, maybe if a
>>>> db-name starts with _ we should call the lost and found a_users_lost+found
>>>> (_ sorts at the top, so "a" will be near it and legal).
>>>>
>>>>
>>>>
>>>> When a database has readers defined in the security object, the tool is
>>>> unable to open them (the reading part of the repair tool needs to have the
>>>> _admin userCtx, not just the writer).
>>>>
>>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined} vs
>>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>>> escript: exception throw: {unauthorized,<<"You are not authorized to access
>>>> this db.">>}
>>>> in function couch_db:open/2
>>>> in call from couch_db_repair:make_lost_and_found/3
>>>> in call from recover_couchdb:main/1
>>>> in call from escript:run/2
>>>> in call from escript:start/1
>>>> in call from init:start_it/1
>>>> in call from init:start_em/1
>>>>
>>>>
>>>> It would also be helpful if the status lines could say something more than
>>>>
>>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to bench_lost+found
>>>>
>>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>>
>>>>
>>>> I will patch the first few, I'd love help from someone on the last one.
>>>> I'll be on IRC.
>>>>
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>>
>>>>>
>>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>>
>>>>>> Hi, Jason.
>>>>>>
>>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>>
>>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>>> that he
>>>>>>>> had packaged the whole thing up as an escript + some .beams. If we
>>>> can get
>>>>>>>> it down to a single file a la rebar that would be a pretty sweet way
>>>> to
>>>>>>>> deliver the repair tool in my opinion.
>>>>>>>>
>>>>>>>
>>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>>
>>>>>>
>>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>>
>>>>>
>>>>> I think it is important that we package and release this, if it is ready.
>>>> We should link to it from the bug description page, the project home page,
>>>> as well as blog about it, etc. What is the point of working feverishly on a
>>>> recovery tool if we don't go the last mile?
>>>>>
>>>>> I am testing it now on my database directory to make sure it doesn't harm
>>>> anything (I was never subject to the bug, which is probably where most
>>>> people are, but they might run it anyway.)
>>>>>
>>>>> As it stands the submodules thing can't be part of the release, we need
>>>> to package it up as a single zip file or something.
>>>>>
>>>>> Is there anything else that needs to be done before we can release this?
>>>>>
>>>>> Chris
>>>>>
>>>>>> --
>>>>>> Jason Smith
>>>>>> Couchio Hosting
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Jason Smith
>>> Couchio Hosting
>>
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 12, 2010, at 12:36 PM, Adam Kocoloski wrote:
> Right, and jchris' db_repair branch includes my patches for DB reader _admin access and a more useful progress report in the replication phase of the repair.
>
I've updated the repair branch with everyone's code. I think it is faster, due to Adam's idea that if we run the merges in reverse order, those near the front of the file are more likely to be no-ops, so less work is done over all.
Mikeal will be testing for correctness. Could other's please use it and test for usability as well. Latest code (with instructions) is here:
http://github.com/jhs/recover-couchdb/
Which points at http://github.com/jchris/couchdb/tree/db_repair for the repair code.
One thing I am not clear about (need better docs) is, do we need to replicate the original db to the lost+found db (or vice-versa), after recovery is complete?
Chris
> Adam
>
> On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
>
>> The code is updated with the following changes:
>> 1. Adhere to the lost+found/databasename custom...
>> 2. ...except databases starting with _, which goes into
>> _system/databasename
>> 3. Sync up with jchris's db_repair branch
>>
>> (About #2, I started with _/database but I think it's too easy to miss at
>> the command line.)
>>
>> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com> wrote:
>>
>>> A few bug reports from my testing:
>>>
>>> I launched with this command, as specified in the README:
>>>
>>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec ./recover_couchdb
>>> {} \;
>>>
>>>
>>>
>>> First of all, it chokes on my _users and _replicator db:
>>>
>>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at 0
>>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>>> {error,illegal_database_name}}
>>>
>>> That second [error] line is repeated many many times (once per merge I
>>> think). I think the issue is that _users is hard-coded to be OK, but
>>> _users_lost+found is not. So we should do something about that, maybe if a
>>> db-name starts with _ we should call the lost and found a_users_lost+found
>>> (_ sorts at the top, so "a" will be near it and legal).
>>>
>>>
>>>
>>> When a database has readers defined in the security object, the tool is
>>> unable to open them (the reading part of the repair tool needs to have the
>>> _admin userCtx, not just the writer).
>>>
>>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined} vs
>>> Names [<<"joe">>] Roles [<<"_admin">>]
>>> escript: exception throw: {unauthorized,<<"You are not authorized to access
>>> this db.">>}
>>> in function couch_db:open/2
>>> in call from couch_db_repair:make_lost_and_found/3
>>> in call from recover_couchdb:main/1
>>> in call from escript:run/2
>>> in call from escript:start/1
>>> in call from init:start_it/1
>>> in call from init:start_em/1
>>>
>>>
>>> It would also be helpful if the status lines could say something more than
>>>
>>> [info] [<0.2.0>] couch_db_repair writing 15 updates to bench_lost+found
>>>
>>> Like maybe add a note like "about 23% complete" if at all possible.
>>>
>>>
>>> I will patch the first few, I'd love help from someone on the last one.
>>> I'll be on IRC.
>>>
>>>
>>> Cheers,
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>>
>>>>
>>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>>
>>>>> Hi, Jason.
>>>>>
>>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>>
>>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org>
>>> wrote:
>>>>>>
>>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>>> that he
>>>>>>> had packaged the whole thing up as an escript + some .beams. If we
>>> can get
>>>>>>> it down to a single file a la rebar that would be a pretty sweet way
>>> to
>>>>>>> deliver the repair tool in my opinion.
>>>>>>>
>>>>>>
>>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>>
>>>>>
>>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>>
>>>>
>>>> I think it is important that we package and release this, if it is ready.
>>> We should link to it from the bug description page, the project home page,
>>> as well as blog about it, etc. What is the point of working feverishly on a
>>> recovery tool if we don't go the last mile?
>>>>
>>>> I am testing it now on my database directory to make sure it doesn't harm
>>> anything (I was never subject to the bug, which is probably where most
>>> people are, but they might run it anyway.)
>>>>
>>>> As it stands the submodules thing can't be part of the release, we need
>>> to package it up as a single zip file or something.
>>>>
>>>> Is there anything else that needs to be done before we can release this?
>>>>
>>>> Chris
>>>>
>>>>> --
>>>>> Jason Smith
>>>>> Couchio Hosting
>>>>
>>>
>>>
>>
>>
>> --
>> Jason Smith
>> Couchio Hosting
>
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
Right, and jchris' db_repair branch includes my patches for DB reader _admin access and a more useful progress report in the replication phase of the repair.
Adam
On Aug 12, 2010, at 3:14 PM, Jason Smith wrote:
> The code is updated with the following changes:
> 1. Adhere to the lost+found/databasename custom...
> 2. ...except databases starting with _, which goes into
> _system/databasename
> 3. Sync up with jchris's db_repair branch
>
> (About #2, I started with _/database but I think it's too easy to miss at
> the command line.)
>
> On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com> wrote:
>
>> A few bug reports from my testing:
>>
>> I launched with this command, as specified in the README:
>>
>> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec ./recover_couchdb
>> {} \;
>>
>>
>>
>> First of all, it chokes on my _users and _replicator db:
>>
>> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at 0
>> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
>> {error,illegal_database_name}}
>>
>> That second [error] line is repeated many many times (once per merge I
>> think). I think the issue is that _users is hard-coded to be OK, but
>> _users_lost+found is not. So we should do something about that, maybe if a
>> db-name starts with _ we should call the lost and found a_users_lost+found
>> (_ sorts at the top, so "a" will be near it and legal).
>>
>>
>>
>> When a database has readers defined in the security object, the tool is
>> unable to open them (the reading part of the repair tool needs to have the
>> _admin userCtx, not just the writer).
>>
>> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined} vs
>> Names [<<"joe">>] Roles [<<"_admin">>]
>> escript: exception throw: {unauthorized,<<"You are not authorized to access
>> this db.">>}
>> in function couch_db:open/2
>> in call from couch_db_repair:make_lost_and_found/3
>> in call from recover_couchdb:main/1
>> in call from escript:run/2
>> in call from escript:start/1
>> in call from init:start_it/1
>> in call from init:start_em/1
>>
>>
>> It would also be helpful if the status lines could say something more than
>>
>> [info] [<0.2.0>] couch_db_repair writing 15 updates to bench_lost+found
>>
>> Like maybe add a note like "about 23% complete" if at all possible.
>>
>>
>> I will patch the first few, I'd love help from someone on the last one.
>> I'll be on IRC.
>>
>>
>> Cheers,
>> Chris
>>
>>
>>
>>
>>
>>
>>
>> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>>
>>>
>>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>>
>>>> Hi, Jason.
>>>>
>>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>>
>>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org>
>> wrote:
>>>>>
>>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
>> that he
>>>>>> had packaged the whole thing up as an escript + some .beams. If we
>> can get
>>>>>> it down to a single file a la rebar that would be a pretty sweet way
>> to
>>>>>> deliver the repair tool in my opinion.
>>>>>>
>>>>>
>>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>>
>>>>
>>>> I think you mean http://github.com/jhs/recover-couchdb
>>>>
>>>
>>> I think it is important that we package and release this, if it is ready.
>> We should link to it from the bug description page, the project home page,
>> as well as blog about it, etc. What is the point of working feverishly on a
>> recovery tool if we don't go the last mile?
>>>
>>> I am testing it now on my database directory to make sure it doesn't harm
>> anything (I was never subject to the bug, which is probably where most
>> people are, but they might run it anyway.)
>>>
>>> As it stands the submodules thing can't be part of the release, we need
>> to package it up as a single zip file or something.
>>>
>>> Is there anything else that needs to be done before we can release this?
>>>
>>> Chris
>>>
>>>> --
>>>> Jason Smith
>>>> Couchio Hosting
>>>
>>
>>
>
>
> --
> Jason Smith
> Couchio Hosting
Re: data recovery tool progress
Posted by Jason Smith <jh...@couch.io>.
The code is updated with the following changes:
1. Adhere to the lost+found/databasename custom...
2. ...except databases starting with _, which goes into
_system/databasename
3. Sync up with jchris's db_repair branch
(About #2, I started with _/database but I think it's too easy to miss at
the command line.)
On Fri, Aug 13, 2010 at 00:52, J Chris Anderson <jc...@gmail.com> wrote:
> A few bug reports from my testing:
>
> I launched with this command, as specified in the README:
>
> find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec ./recover_couchdb
> {} \;
>
>
>
> First of all, it chokes on my _users and _replicator db:
>
> [info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at 0
> [error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
> {error,illegal_database_name}}
>
> That second [error] line is repeated many many times (once per merge I
> think). I think the issue is that _users is hard-coded to be OK, but
> _users_lost+found is not. So we should do something about that, maybe if a
> db-name starts with _ we should call the lost and found a_users_lost+found
> (_ sorts at the top, so "a" will be near it and legal).
>
>
>
> When a database has readers defined in the security object, the tool is
> unable to open them (the reading part of the repair tool needs to have the
> _admin userCtx, not just the writer).
>
> [debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined} vs
> Names [<<"joe">>] Roles [<<"_admin">>]
> escript: exception throw: {unauthorized,<<"You are not authorized to access
> this db.">>}
> in function couch_db:open/2
> in call from couch_db_repair:make_lost_and_found/3
> in call from recover_couchdb:main/1
> in call from escript:run/2
> in call from escript:start/1
> in call from init:start_it/1
> in call from init:start_em/1
>
>
> It would also be helpful if the status lines could say something more than
>
> [info] [<0.2.0>] couch_db_repair writing 15 updates to bench_lost+found
>
> Like maybe add a note like "about 23% complete" if at all possible.
>
>
> I will patch the first few, I'd love help from someone on the last one.
> I'll be on IRC.
>
>
> Cheers,
> Chris
>
>
>
>
>
>
>
> On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>
> >
> > On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
> >
> >> Hi, Jason.
> >>
> >> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
> >>
> >>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org>
> wrote:
> >>>
> >>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC
> that he
> >>>> had packaged the whole thing up as an escript + some .beams. If we
> can get
> >>>> it down to a single file a la rebar that would be a pretty sweet way
> to
> >>>> deliver the repair tool in my opinion.
> >>>>
> >>>
> >>> Please check out http://github.com/jhs/repair-couchdb
> >>>
> >>
> >> I think you mean http://github.com/jhs/recover-couchdb
> >>
> >
> > I think it is important that we package and release this, if it is ready.
> We should link to it from the bug description page, the project home page,
> as well as blog about it, etc. What is the point of working feverishly on a
> recovery tool if we don't go the last mile?
> >
> > I am testing it now on my database directory to make sure it doesn't harm
> anything (I was never subject to the bug, which is probably where most
> people are, but they might run it anyway.)
> >
> > As it stands the submodules thing can't be part of the release, we need
> to package it up as a single zip file or something.
> >
> > Is there anything else that needs to be done before we can release this?
> >
> > Chris
> >
> >> --
> >> Jason Smith
> >> Couchio Hosting
> >
>
>
--
Jason Smith
Couchio Hosting
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@gmail.com>.
A few bug reports from my testing:
I launched with this command, as specified in the README:
find ~/code/couchdb/tmp/lib -type f -name '*.couch' -exec ./recover_couchdb {} \;
First of all, it chokes on my _users and _replicator db:
[info] [<0.2.0>] couch_db_repair for _users - scanning 335961 bytes at 0
[error] [<0.2.0>] couch_db_repair merge node at 332061 {case_clause,
{error,illegal_database_name}}
That second [error] line is repeated many many times (once per merge I think). I think the issue is that _users is hard-coded to be OK, but _users_lost+found is not. So we should do something about that, maybe if a db-name starts with _ we should call the lost and found a_users_lost+found (_ sorts at the top, so "a" will be near it and legal).
When a database has readers defined in the security object, the tool is unable to open them (the reading part of the repair tool needs to have the _admin userCtx, not just the writer).
[debug] [<0.2.0>] Not a reader: UserCtx {user_ctx,null,[],undefined} vs Names [<<"joe">>] Roles [<<"_admin">>]
escript: exception throw: {unauthorized,<<"You are not authorized to access this db.">>}
in function couch_db:open/2
in call from couch_db_repair:make_lost_and_found/3
in call from recover_couchdb:main/1
in call from escript:run/2
in call from escript:start/1
in call from init:start_it/1
in call from init:start_em/1
It would also be helpful if the status lines could say something more than
[info] [<0.2.0>] couch_db_repair writing 15 updates to bench_lost+found
Like maybe add a note like "about 23% complete" if at all possible.
I will patch the first few, I'd love help from someone on the last one. I'll be on IRC.
Cheers,
Chris
On Aug 12, 2010, at 10:18 AM, J Chris Anderson wrote:
>
> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>
>> Hi, Jason.
>>
>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>
>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
>>>
>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
>>>> had packaged the whole thing up as an escript + some .beams. If we can get
>>>> it down to a single file a la rebar that would be a pretty sweet way to
>>>> deliver the repair tool in my opinion.
>>>>
>>>
>>> Please check out http://github.com/jhs/repair-couchdb
>>>
>>
>> I think you mean http://github.com/jhs/recover-couchdb
>>
>
> I think it is important that we package and release this, if it is ready. We should link to it from the bug description page, the project home page, as well as blog about it, etc. What is the point of working feverishly on a recovery tool if we don't go the last mile?
>
> I am testing it now on my database directory to make sure it doesn't harm anything (I was never subject to the bug, which is probably where most people are, but they might run it anyway.)
>
> As it stands the submodules thing can't be part of the release, we need to package it up as a single zip file or something.
>
> Is there anything else that needs to be done before we can release this?
>
> Chris
>
>> --
>> Jason Smith
>> Couchio Hosting
>
Re: data recovery tool progress
Posted by Noah Slater <ns...@apache.org>.
I would like to include a link to the final product in the 1.0.1 release announcement.
I estimate that will be going out on Monday evening.
On 12 Aug 2010, at 18:18, J Chris Anderson wrote:
>
> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>
>> Hi, Jason.
>>
>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>
>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
>>>
>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
>>>> had packaged the whole thing up as an escript + some .beams. If we can get
>>>> it down to a single file a la rebar that would be a pretty sweet way to
>>>> deliver the repair tool in my opinion.
>>>>
>>>
>>> Please check out http://github.com/jhs/repair-couchdb
>>>
>>
>> I think you mean http://github.com/jhs/recover-couchdb
>>
>
> I think it is important that we package and release this, if it is ready. We should link to it from the bug description page, the project home page, as well as blog about it, etc. What is the point of working feverishly on a recovery tool if we don't go the last mile?
>
> I am testing it now on my database directory to make sure it doesn't harm anything (I was never subject to the bug, which is probably where most people are, but they might run it anyway.)
>
> As it stands the submodules thing can't be part of the release, we need to package it up as a single zip file or something.
>
> Is there anything else that needs to be done before we can release this?
>
> Chris
>
>> --
>> Jason Smith
>> Couchio Hosting
>
Re: data recovery tool progress
Posted by Noah Slater <ns...@apache.org>.
The last thing we need is a data recovery tool that trashes data.
It's REALLY important we get this right!
/me cheers from the sidelines ;)
On 12 Aug 2010, at 18:47, Robert Dionne wrote:
> I download Jason's escripts, ran the make, and tested against the 1.0.0 release, created a few bad dbs and also tested against a couple of good ones. It all looked good on OS X.
>
> The final solution ended up to be only ~118 lines of code, which I spent a couple of hours looking over, to check as well as to refresh my memory of couchdb internals. There's several dead functions in the file I'd recommend removing, or commenting out, as some users may want to read the code.
>
> It's probably already been done by all the committers, as well as others, and I suspect a few have it burnt into their brain's ram by now, but I'd also suggest as many people as possible who know the internals take a hard look at those 118 lines. You know, eyeballs, bla...
>
>
>
>
>
> On Aug 12, 2010, at 1:18 PM, J Chris Anderson wrote:
>
>>
>> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>>
>>> Hi, Jason.
>>>
>>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>>
>>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
>>>>
>>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
>>>>> had packaged the whole thing up as an escript + some .beams. If we can get
>>>>> it down to a single file a la rebar that would be a pretty sweet way to
>>>>> deliver the repair tool in my opinion.
>>>>>
>>>>
>>>> Please check out http://github.com/jhs/repair-couchdb
>>>>
>>>
>>> I think you mean http://github.com/jhs/recover-couchdb
>>>
>>
>> I think it is important that we package and release this, if it is ready. We should link to it from the bug description page, the project home page, as well as blog about it, etc. What is the point of working feverishly on a recovery tool if we don't go the last mile?
>>
>> I am testing it now on my database directory to make sure it doesn't harm anything (I was never subject to the bug, which is probably where most people are, but they might run it anyway.)
>>
>> As it stands the submodules thing can't be part of the release, we need to package it up as a single zip file or something.
>>
>> Is there anything else that needs to be done before we can release this?
>>
>> Chris
>>
>>> --
>>> Jason Smith
>>> Couchio Hosting
>>
>
Re: data recovery tool progress
Posted by Robert Dionne <di...@dionne-associates.com>.
I download Jason's escripts, ran the make, and tested against the 1.0.0 release, created a few bad dbs and also tested against a couple of good ones. It all looked good on OS X.
The final solution ended up to be only ~118 lines of code, which I spent a couple of hours looking over, to check as well as to refresh my memory of couchdb internals. There's several dead functions in the file I'd recommend removing, or commenting out, as some users may want to read the code.
It's probably already been done by all the committers, as well as others, and I suspect a few have it burnt into their brain's ram by now, but I'd also suggest as many people as possible who know the internals take a hard look at those 118 lines. You know, eyeballs, bla...
On Aug 12, 2010, at 1:18 PM, J Chris Anderson wrote:
>
> On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
>
>> Hi, Jason.
>>
>> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>>
>>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
>>>
>>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
>>>> had packaged the whole thing up as an escript + some .beams. If we can get
>>>> it down to a single file a la rebar that would be a pretty sweet way to
>>>> deliver the repair tool in my opinion.
>>>>
>>>
>>> Please check out http://github.com/jhs/repair-couchdb
>>>
>>
>> I think you mean http://github.com/jhs/recover-couchdb
>>
>
> I think it is important that we package and release this, if it is ready. We should link to it from the bug description page, the project home page, as well as blog about it, etc. What is the point of working feverishly on a recovery tool if we don't go the last mile?
>
> I am testing it now on my database directory to make sure it doesn't harm anything (I was never subject to the bug, which is probably where most people are, but they might run it anyway.)
>
> As it stands the submodules thing can't be part of the release, we need to package it up as a single zip file or something.
>
> Is there anything else that needs to be done before we can release this?
>
> Chris
>
>> --
>> Jason Smith
>> Couchio Hosting
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 11, 2010, at 2:14 PM, Jason Smith wrote:
> Hi, Jason.
>
> On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
>
>> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
>>> had packaged the whole thing up as an escript + some .beams. If we can get
>>> it down to a single file a la rebar that would be a pretty sweet way to
>>> deliver the repair tool in my opinion.
>>>
>>
>> Please check out http://github.com/jhs/repair-couchdb
>>
>
> I think you mean http://github.com/jhs/recover-couchdb
>
I think it is important that we package and release this, if it is ready. We should link to it from the bug description page, the project home page, as well as blog about it, etc. What is the point of working feverishly on a recovery tool if we don't go the last mile?
I am testing it now on my database directory to make sure it doesn't harm anything (I was never subject to the bug, which is probably where most people are, but they might run it anyway.)
As it stands the submodules thing can't be part of the release, we need to package it up as a single zip file or something.
Is there anything else that needs to be done before we can release this?
Chris
> --
> Jason Smith
> Couchio Hosting
Re: data recovery tool progress
Posted by Jason Smith <jh...@couch.io>.
Hi, Jason.
On Thu, Aug 12, 2010 at 04:14, Jason Smith <jh...@couch.io> wrote:
> On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
>
>> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
>> had packaged the whole thing up as an escript + some .beams. If we can get
>> it down to a single file a la rebar that would be a pretty sweet way to
>> deliver the repair tool in my opinion.
>>
>
> Please check out http://github.com/jhs/repair-couchdb
>
I think you mean http://github.com/jhs/recover-couchdb
--
Jason Smith
Couchio Hosting
Re: data recovery tool progress
Posted by Jason Smith <jh...@couch.io>.
On Wed, Aug 11, 2010 at 09:52, Adam Kocoloski <ko...@apache.org> wrote:
> Excellent, thanks for testing. I caught Jason Smith saying on IRC that he
> had packaged the whole thing up as an escript + some .beams. If we can get
> it down to a single file a la rebar that would be a pretty sweet way to
> deliver the repair tool in my opinion.
>
Please check out http://github.com/jhs/repair-couchdb
It uses Git submodules (quiet down!) and builds Adam's latest code into an
escript about 260KB. It runs great for me on 32 and 64 bit Linux and OSX.
The only notable difference is I chose slightly different names for
lost+found for reasons I don't remember anymore.
I would like for people to test it, especially on Windows which I have not
done yet.
Of course if there are bugs please let the list know; or if you think more
features are warranted (e.g. wildcards) then let's discuss that too.
If it looks good we can tell people to download the escript and fire away.
--
Jason Smith
Couchio Hosting
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
Excellent, thanks for testing. I caught Jason Smith saying on IRC that he had packaged the whole thing up as an escript + some .beams. If we can get it down to a single file a la rebar that would be a pretty sweet way to deliver the repair tool in my opinion.
Adam
On Aug 10, 2010, at 10:40 PM, Mikeal Rogers wrote:
> Ok, latest code has been tested against every db that I have and it works
> great.
>
> What are our next steps here?
>
> I'd like to get this out to all the people who didn't feel comfortable send
> me their db to test against before we release it more widely.
>
> -Mikeal
>
> On Tue, Aug 10, 2010 at 6:11 PM, Mikeal Rogers <mi...@gmail.com>wrote:
>
>> Found one issue, we weren't picking up design docs because it didn't have
>> admin privileges.
>>
>> Adam fixed it and pushed and I've verified that it works now.
>>
>> I wrote a little node script to show all recovered documents and expose any
>> documents that didn't make it in to lost+found.
>>
>> http://github.com/mikeal/couchtest/blob/master/validate.js
>>
>> Requires request, `npm install request`.
>>
>> I'm now running recover on all the test db's I have and running the
>> validation script against them.
>>
>> -Mikeal
>>
>>
>> On Tue, Aug 10, 2010 at 1:34 PM, Mikeal Rogers <mi...@gmail.com>wrote:
>>
>>> I have some timing number for the new code.
>>>
>>> multi_conflict has 200 lost documents and 201 documents total after
>>> recovery.
>>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
>>> {25217069,ok}
>>> 25 seconds
>>>
>>> Something funky is going on here. Investigating.
>>> 1> timer:tc(couch_db_repair, make_lost_and_found,
>>> ["multi_conflict_with_attach"]).
>>> {654782,ok}
>>> .6 seconds
>>>
>>> This db has 124969 documents in it.
>>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["testwritesdb"]).
>>> {1381969304,ok}
>>> 23 minutes
>>>
>>> This database is about 500megs and 46660 before recovery and 46801 after.
>>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["prod"]).
>>> {2329669113,ok}
>>> 38.8 minutes
>>>
>>> -Mikeal
>>>
>>> On Tue, Aug 10, 2010 at 12:06 PM, Adam Kocoloski <ko...@apache.org>wrote:
>>>
>>>> Good idea. Now we've got
>>>>
>>>>> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
>>>> bytes at 1380102
>>>>> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
>>>> bytes at 331526
>>>>> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 331526
>>>> bytes at 0
>>>>> [info] [<0.33.0>] couch_db_repair writing 12 updates to
>>>> lost+found/testwritesdb
>>>>> [info] [<0.33.0>] couch_db_repair writing 9 updates to
>>>> lost+found/testwritesdb
>>>>> [info] [<0.33.0>] couch_db_repair writing 8 updates to
>>>> lost+found/testwritesdb
>>>>
>>>> Adam
>>>>
>>>> On Aug 10, 2010, at 2:29 PM, Robert Newson wrote:
>>>>
>>>>> It took 20 minutes before the first 'update' line came out, but now
>>>>> seems to be recovering smoothly. machine load is back down to sane
>>>>> levels.
>>>>>
>>>>> Suggest feedback during the hunting phase.
>>>>>
>>>>> B.
>>>>>
>>>>> On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <ko...@apache.org>
>>>> wrote:
>>>>>> Thanks for the crosscheck. I'm not aware of anything in the node
>>>> finder that would cause it to struggle mightily with healthy DBs. It pretty
>>>> much ignores the health of the DB, in fact. Would be interested to hear
>>>> more.
>>>>>>
>>>>>> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
>>>>>>
>>>>>>> I verified the new code's ability to repair the testwritesdb. system
>>>>>>> load was smooth from start to finish.
>>>>>>>
>>>>>>> I started a further test on a different (healthy) database and system
>>>>>>> load was severe again, just collecting the roots (the lost+found db
>>>>>>> was not yet created when I aborted the attempt). I suspect the fact
>>>>>>> that it's healthy is the issue, so if I'm right, perhaps a warning is
>>>>>>> useful.
>>>>>>>
>>>>>>> B.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org>
>>>> wrote:
>>>>>>>> Another update. This morning I took a different tack and, rather
>>>> than try to find root nodes, I just looked for all kv_nodes in the file and
>>>> treated each of those as a separate virtual DB to be replicated. This
>>>> reduces the algorithmic complexity of the repair, and it looks like
>>>> testwritesdb repairs in ~30 minutes or so. Also, this method results in the
>>>> lost+found DB containing every document, not just the missing ones.
>>>>>>>>
>>>>>>>> My branch does not currently include Randall's parallelization of
>>>> the replications. It's still CPU-limited, so that may be a worthwhile
>>>> optimization. On the other hand, I think we may be reaching a stage at
>>>> which performance for this repair tool is 'good enough', and pmaps can make
>>>> error handling a bit dicey.
>>>>>>>>
>>>>>>>> In short, I think this tool is now in good shape.
>>>>>>>>
>>>>>>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
Ok, latest code has been tested against every db that I have and it works
great.
What are our next steps here?
I'd like to get this out to all the people who didn't feel comfortable send
me their db to test against before we release it more widely.
-Mikeal
On Tue, Aug 10, 2010 at 6:11 PM, Mikeal Rogers <mi...@gmail.com>wrote:
> Found one issue, we weren't picking up design docs because it didn't have
> admin privileges.
>
> Adam fixed it and pushed and I've verified that it works now.
>
> I wrote a little node script to show all recovered documents and expose any
> documents that didn't make it in to lost+found.
>
> http://github.com/mikeal/couchtest/blob/master/validate.js
>
> Requires request, `npm install request`.
>
> I'm now running recover on all the test db's I have and running the
> validation script against them.
>
> -Mikeal
>
>
> On Tue, Aug 10, 2010 at 1:34 PM, Mikeal Rogers <mi...@gmail.com>wrote:
>
>> I have some timing number for the new code.
>>
>> multi_conflict has 200 lost documents and 201 documents total after
>> recovery.
>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
>> {25217069,ok}
>> 25 seconds
>>
>> Something funky is going on here. Investigating.
>> 1> timer:tc(couch_db_repair, make_lost_and_found,
>> ["multi_conflict_with_attach"]).
>> {654782,ok}
>> .6 seconds
>>
>> This db has 124969 documents in it.
>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["testwritesdb"]).
>> {1381969304,ok}
>> 23 minutes
>>
>> This database is about 500megs and 46660 before recovery and 46801 after.
>> 1> timer:tc(couch_db_repair, make_lost_and_found, ["prod"]).
>> {2329669113,ok}
>> 38.8 minutes
>>
>> -Mikeal
>>
>> On Tue, Aug 10, 2010 at 12:06 PM, Adam Kocoloski <ko...@apache.org>wrote:
>>
>>> Good idea. Now we've got
>>>
>>> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
>>> bytes at 1380102
>>> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
>>> bytes at 331526
>>> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 331526
>>> bytes at 0
>>> > [info] [<0.33.0>] couch_db_repair writing 12 updates to
>>> lost+found/testwritesdb
>>> > [info] [<0.33.0>] couch_db_repair writing 9 updates to
>>> lost+found/testwritesdb
>>> > [info] [<0.33.0>] couch_db_repair writing 8 updates to
>>> lost+found/testwritesdb
>>>
>>> Adam
>>>
>>> On Aug 10, 2010, at 2:29 PM, Robert Newson wrote:
>>>
>>> > It took 20 minutes before the first 'update' line came out, but now
>>> > seems to be recovering smoothly. machine load is back down to sane
>>> > levels.
>>> >
>>> > Suggest feedback during the hunting phase.
>>> >
>>> > B.
>>> >
>>> > On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <ko...@apache.org>
>>> wrote:
>>> >> Thanks for the crosscheck. I'm not aware of anything in the node
>>> finder that would cause it to struggle mightily with healthy DBs. It pretty
>>> much ignores the health of the DB, in fact. Would be interested to hear
>>> more.
>>> >>
>>> >> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
>>> >>
>>> >>> I verified the new code's ability to repair the testwritesdb. system
>>> >>> load was smooth from start to finish.
>>> >>>
>>> >>> I started a further test on a different (healthy) database and system
>>> >>> load was severe again, just collecting the roots (the lost+found db
>>> >>> was not yet created when I aborted the attempt). I suspect the fact
>>> >>> that it's healthy is the issue, so if I'm right, perhaps a warning is
>>> >>> useful.
>>> >>>
>>> >>> B.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org>
>>> wrote:
>>> >>>> Another update. This morning I took a different tack and, rather
>>> than try to find root nodes, I just looked for all kv_nodes in the file and
>>> treated each of those as a separate virtual DB to be replicated. This
>>> reduces the algorithmic complexity of the repair, and it looks like
>>> testwritesdb repairs in ~30 minutes or so. Also, this method results in the
>>> lost+found DB containing every document, not just the missing ones.
>>> >>>>
>>> >>>> My branch does not currently include Randall's parallelization of
>>> the replications. It's still CPU-limited, so that may be a worthwhile
>>> optimization. On the other hand, I think we may be reaching a stage at
>>> which performance for this repair tool is 'good enough', and pmaps can make
>>> error handling a bit dicey.
>>> >>>>
>>> >>>> In short, I think this tool is now in good shape.
>>> >>>>
>>> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>> >>>>
>>> >>
>>> >>
>>>
>>>
>>
>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
Found one issue, we weren't picking up design docs because it didn't have
admin privileges.
Adam fixed it and pushed and I've verified that it works now.
I wrote a little node script to show all recovered documents and expose any
documents that didn't make it in to lost+found.
http://github.com/mikeal/couchtest/blob/master/validate.js
Requires request, `npm install request`.
I'm now running recover on all the test db's I have and running the
validation script against them.
-Mikeal
On Tue, Aug 10, 2010 at 1:34 PM, Mikeal Rogers <mi...@gmail.com>wrote:
> I have some timing number for the new code.
>
> multi_conflict has 200 lost documents and 201 documents total after
> recovery.
> 1> timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
> {25217069,ok}
> 25 seconds
>
> Something funky is going on here. Investigating.
> 1> timer:tc(couch_db_repair, make_lost_and_found,
> ["multi_conflict_with_attach"]).
> {654782,ok}
> .6 seconds
>
> This db has 124969 documents in it.
> 1> timer:tc(couch_db_repair, make_lost_and_found, ["testwritesdb"]).
> {1381969304,ok}
> 23 minutes
>
> This database is about 500megs and 46660 before recovery and 46801 after.
> 1> timer:tc(couch_db_repair, make_lost_and_found, ["prod"]).
> {2329669113,ok}
> 38.8 minutes
>
> -Mikeal
>
> On Tue, Aug 10, 2010 at 12:06 PM, Adam Kocoloski <ko...@apache.org>wrote:
>
>> Good idea. Now we've got
>>
>> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
>> bytes at 1380102
>> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
>> bytes at 331526
>> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 331526
>> bytes at 0
>> > [info] [<0.33.0>] couch_db_repair writing 12 updates to
>> lost+found/testwritesdb
>> > [info] [<0.33.0>] couch_db_repair writing 9 updates to
>> lost+found/testwritesdb
>> > [info] [<0.33.0>] couch_db_repair writing 8 updates to
>> lost+found/testwritesdb
>>
>> Adam
>>
>> On Aug 10, 2010, at 2:29 PM, Robert Newson wrote:
>>
>> > It took 20 minutes before the first 'update' line came out, but now
>> > seems to be recovering smoothly. machine load is back down to sane
>> > levels.
>> >
>> > Suggest feedback during the hunting phase.
>> >
>> > B.
>> >
>> > On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <ko...@apache.org>
>> wrote:
>> >> Thanks for the crosscheck. I'm not aware of anything in the node
>> finder that would cause it to struggle mightily with healthy DBs. It pretty
>> much ignores the health of the DB, in fact. Would be interested to hear
>> more.
>> >>
>> >> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
>> >>
>> >>> I verified the new code's ability to repair the testwritesdb. system
>> >>> load was smooth from start to finish.
>> >>>
>> >>> I started a further test on a different (healthy) database and system
>> >>> load was severe again, just collecting the roots (the lost+found db
>> >>> was not yet created when I aborted the attempt). I suspect the fact
>> >>> that it's healthy is the issue, so if I'm right, perhaps a warning is
>> >>> useful.
>> >>>
>> >>> B.
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org>
>> wrote:
>> >>>> Another update. This morning I took a different tack and, rather
>> than try to find root nodes, I just looked for all kv_nodes in the file and
>> treated each of those as a separate virtual DB to be replicated. This
>> reduces the algorithmic complexity of the repair, and it looks like
>> testwritesdb repairs in ~30 minutes or so. Also, this method results in the
>> lost+found DB containing every document, not just the missing ones.
>> >>>>
>> >>>> My branch does not currently include Randall's parallelization of the
>> replications. It's still CPU-limited, so that may be a worthwhile
>> optimization. On the other hand, I think we may be reaching a stage at
>> which performance for this repair tool is 'good enough', and pmaps can make
>> error handling a bit dicey.
>> >>>>
>> >>>> In short, I think this tool is now in good shape.
>> >>>>
>> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
>> >>>>
>> >>
>> >>
>>
>>
>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
I have some timing number for the new code.
multi_conflict has 200 lost documents and 201 documents total after
recovery.
1> timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
{25217069,ok}
25 seconds
Something funky is going on here. Investigating.
1> timer:tc(couch_db_repair, make_lost_and_found,
["multi_conflict_with_attach"]).
{654782,ok}
.6 seconds
This db has 124969 documents in it.
1> timer:tc(couch_db_repair, make_lost_and_found, ["testwritesdb"]).
{1381969304,ok}
23 minutes
This database is about 500megs and 46660 before recovery and 46801 after.
1> timer:tc(couch_db_repair, make_lost_and_found, ["prod"]).
{2329669113,ok}
38.8 minutes
-Mikeal
On Tue, Aug 10, 2010 at 12:06 PM, Adam Kocoloski <ko...@apache.org>wrote:
> Good idea. Now we've got
>
> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
> bytes at 1380102
> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576
> bytes at 331526
> > [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 331526
> bytes at 0
> > [info] [<0.33.0>] couch_db_repair writing 12 updates to
> lost+found/testwritesdb
> > [info] [<0.33.0>] couch_db_repair writing 9 updates to
> lost+found/testwritesdb
> > [info] [<0.33.0>] couch_db_repair writing 8 updates to
> lost+found/testwritesdb
>
> Adam
>
> On Aug 10, 2010, at 2:29 PM, Robert Newson wrote:
>
> > It took 20 minutes before the first 'update' line came out, but now
> > seems to be recovering smoothly. machine load is back down to sane
> > levels.
> >
> > Suggest feedback during the hunting phase.
> >
> > B.
> >
> > On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <ko...@apache.org>
> wrote:
> >> Thanks for the crosscheck. I'm not aware of anything in the node finder
> that would cause it to struggle mightily with healthy DBs. It pretty much
> ignores the health of the DB, in fact. Would be interested to hear more.
> >>
> >> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
> >>
> >>> I verified the new code's ability to repair the testwritesdb. system
> >>> load was smooth from start to finish.
> >>>
> >>> I started a further test on a different (healthy) database and system
> >>> load was severe again, just collecting the roots (the lost+found db
> >>> was not yet created when I aborted the attempt). I suspect the fact
> >>> that it's healthy is the issue, so if I'm right, perhaps a warning is
> >>> useful.
> >>>
> >>> B.
> >>>
> >>>
> >>>
> >>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org>
> wrote:
> >>>> Another update. This morning I took a different tack and, rather than
> try to find root nodes, I just looked for all kv_nodes in the file and
> treated each of those as a separate virtual DB to be replicated. This
> reduces the algorithmic complexity of the repair, and it looks like
> testwritesdb repairs in ~30 minutes or so. Also, this method results in the
> lost+found DB containing every document, not just the missing ones.
> >>>>
> >>>> My branch does not currently include Randall's parallelization of the
> replications. It's still CPU-limited, so that may be a worthwhile
> optimization. On the other hand, I think we may be reaching a stage at
> which performance for this repair tool is 'good enough', and pmaps can make
> error handling a bit dicey.
> >>>>
> >>>> In short, I think this tool is now in good shape.
> >>>>
> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
> >>>>
> >>
> >>
>
>
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
Good idea. Now we've got
> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576 bytes at 1380102
> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 1048576 bytes at 331526
> [info] [<0.33.0>] couch_db_repair for testwritesdb - scanning 331526 bytes at 0
> [info] [<0.33.0>] couch_db_repair writing 12 updates to lost+found/testwritesdb
> [info] [<0.33.0>] couch_db_repair writing 9 updates to lost+found/testwritesdb
> [info] [<0.33.0>] couch_db_repair writing 8 updates to lost+found/testwritesdb
Adam
On Aug 10, 2010, at 2:29 PM, Robert Newson wrote:
> It took 20 minutes before the first 'update' line came out, but now
> seems to be recovering smoothly. machine load is back down to sane
> levels.
>
> Suggest feedback during the hunting phase.
>
> B.
>
> On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <ko...@apache.org> wrote:
>> Thanks for the crosscheck. I'm not aware of anything in the node finder that would cause it to struggle mightily with healthy DBs. It pretty much ignores the health of the DB, in fact. Would be interested to hear more.
>>
>> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
>>
>>> I verified the new code's ability to repair the testwritesdb. system
>>> load was smooth from start to finish.
>>>
>>> I started a further test on a different (healthy) database and system
>>> load was severe again, just collecting the roots (the lost+found db
>>> was not yet created when I aborted the attempt). I suspect the fact
>>> that it's healthy is the issue, so if I'm right, perhaps a warning is
>>> useful.
>>>
>>> B.
>>>
>>>
>>>
>>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>> Another update. This morning I took a different tack and, rather than try to find root nodes, I just looked for all kv_nodes in the file and treated each of those as a separate virtual DB to be replicated. This reduces the algorithmic complexity of the repair, and it looks like testwritesdb repairs in ~30 minutes or so. Also, this method results in the lost+found DB containing every document, not just the missing ones.
>>>>
>>>> My branch does not currently include Randall's parallelization of the replications. It's still CPU-limited, so that may be a worthwhile optimization. On the other hand, I think we may be reaching a stage at which performance for this repair tool is 'good enough', and pmaps can make error handling a bit dicey.
>>>>
>>>> In short, I think this tool is now in good shape.
>>>>
>>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>>
>>
>>
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
It took 20 minutes before the first 'update' line came out, but now
seems to be recovering smoothly. machine load is back down to sane
levels.
Suggest feedback during the hunting phase.
B.
On Tue, Aug 10, 2010 at 7:11 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Thanks for the crosscheck. I'm not aware of anything in the node finder that would cause it to struggle mightily with healthy DBs. It pretty much ignores the health of the DB, in fact. Would be interested to hear more.
>
> On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
>
>> I verified the new code's ability to repair the testwritesdb. system
>> load was smooth from start to finish.
>>
>> I started a further test on a different (healthy) database and system
>> load was severe again, just collecting the roots (the lost+found db
>> was not yet created when I aborted the attempt). I suspect the fact
>> that it's healthy is the issue, so if I'm right, perhaps a warning is
>> useful.
>>
>> B.
>>
>>
>>
>> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>> Another update. This morning I took a different tack and, rather than try to find root nodes, I just looked for all kv_nodes in the file and treated each of those as a separate virtual DB to be replicated. This reduces the algorithmic complexity of the repair, and it looks like testwritesdb repairs in ~30 minutes or so. Also, this method results in the lost+found DB containing every document, not just the missing ones.
>>>
>>> My branch does not currently include Randall's parallelization of the replications. It's still CPU-limited, so that may be a worthwhile optimization. On the other hand, I think we may be reaching a stage at which performance for this repair tool is 'good enough', and pmaps can make error handling a bit dicey.
>>>
>>> In short, I think this tool is now in good shape.
>>>
>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>
>
>
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
Thanks for the crosscheck. I'm not aware of anything in the node finder that would cause it to struggle mightily with healthy DBs. It pretty much ignores the health of the DB, in fact. Would be interested to hear more.
On Aug 10, 2010, at 1:59 PM, Robert Newson wrote:
> I verified the new code's ability to repair the testwritesdb. system
> load was smooth from start to finish.
>
> I started a further test on a different (healthy) database and system
> load was severe again, just collecting the roots (the lost+found db
> was not yet created when I aborted the attempt). I suspect the fact
> that it's healthy is the issue, so if I'm right, perhaps a warning is
> useful.
>
> B.
>
>
>
> On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org> wrote:
>> Another update. This morning I took a different tack and, rather than try to find root nodes, I just looked for all kv_nodes in the file and treated each of those as a separate virtual DB to be replicated. This reduces the algorithmic complexity of the repair, and it looks like testwritesdb repairs in ~30 minutes or so. Also, this method results in the lost+found DB containing every document, not just the missing ones.
>>
>> My branch does not currently include Randall's parallelization of the replications. It's still CPU-limited, so that may be a worthwhile optimization. On the other hand, I think we may be reaching a stage at which performance for this repair tool is 'good enough', and pmaps can make error handling a bit dicey.
>>
>> In short, I think this tool is now in good shape.
>>
>> http://github.com/kocolosk/couchdb/tree/db_repair
>>
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
I verified the new code's ability to repair the testwritesdb. system
load was smooth from start to finish.
I started a further test on a different (healthy) database and system
load was severe again, just collecting the roots (the lost+found db
was not yet created when I aborted the attempt). I suspect the fact
that it's healthy is the issue, so if I'm right, perhaps a warning is
useful.
B.
On Tue, Aug 10, 2010 at 6:53 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Another update. This morning I took a different tack and, rather than try to find root nodes, I just looked for all kv_nodes in the file and treated each of those as a separate virtual DB to be replicated. This reduces the algorithmic complexity of the repair, and it looks like testwritesdb repairs in ~30 minutes or so. Also, this method results in the lost+found DB containing every document, not just the missing ones.
>
> My branch does not currently include Randall's parallelization of the replications. It's still CPU-limited, so that may be a worthwhile optimization. On the other hand, I think we may be reaching a stage at which performance for this repair tool is 'good enough', and pmaps can make error handling a bit dicey.
>
> In short, I think this tool is now in good shape.
>
> http://github.com/kocolosk/couchdb/tree/db_repair
>
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
Another update. This morning I took a different tack and, rather than try to find root nodes, I just looked for all kv_nodes in the file and treated each of those as a separate virtual DB to be replicated. This reduces the algorithmic complexity of the repair, and it looks like testwritesdb repairs in ~30 minutes or so. Also, this method results in the lost+found DB containing every document, not just the missing ones.
My branch does not currently include Randall's parallelization of the replications. It's still CPU-limited, so that may be a worthwhile optimization. On the other hand, I think we may be reaching a stage at which performance for this repair tool is 'good enough', and pmaps can make error handling a bit dicey.
In short, I think this tool is now in good shape.
http://github.com/kocolosk/couchdb/tree/db_repair
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
IRC heating up again a bit.
Volker and I compared node counts for a DB we had written. Initially his algorithm couch_db_repair_b:repair/1 found 1 more node than my couch_db_repair:find_nodes_quickly/1. This turned out to be a local_btree node, which is suppressed in the latter code.
On my laptop the testwritesdb replicates about 1 node root / sec into lost+found. With 26000+ roots this would take 7+ hours, so I think it's too early to assume that the merger got stuck.
Obviously, this means we need to prune down that set of lost nodes requiring replication. I think the way to do this is to find all DB headers and remove all nodes they point to (directly or indirectly). Anyone up for it?
Adam
Re: data recovery tool progress
Posted by Noah Slater <ns...@apache.org>.
On 10 Aug 2010, at 14:03, Jason Smith wrote:
> 5. I am pulling the mainstream work into a cross-platform escript fsck
> to at least produce good .couch files for people who can not halt
> their 1.0.0 servers at this time.
You're going to make it work with fsck? Awesome.
Re: data recovery tool progress
Posted by Jason Smith <jh...@couch.io>.
IRC is getting quiet so I will wrap up the current state as I understand
it.
1. Everything begins with Filipe's db_repair branch
2. Volker took an independent tack in his db_repair-b branch which was
later merged with the others. However he claims that his code is
detecting node which the mainstream is not so that needs to be confirmed
and resolved.
3. Meanwhile Adam and Randall branched out in another direction
4. Adam's latest branch is working while Randall has parallelization on
top of that, possibly a bit experimental
5. I am pulling the mainstream work into a cross-platform escript fsck
to at least produce good .couch files for people who can not halt
their 1.0.0 servers at this time.
The wiki has a gitk snapshot of all major branches, pardon the size:
http://wiki.couchone.com/page/recovery-tool
--
Jason Smith
Couchio Hosting
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
slight correction, this was with delayed_commits=false. My framework
does a PUT to ensure that on every test run.
B.
On Tue, Aug 10, 2010 at 9:55 AM, Robert Newson <ro...@gmail.com> wrote:
> In ran the db_repair code on a healthy database produced with
> delayed_commits=true.
>
> The source db had 3218 docs. db_repair recovered 3120 and then returned with ok.
>
> I'm redoing that test, but this indicates we're not finding all roots.
>
> I note that the output file was 36 times the input file, which is a
> consequence of folding all possible roots. I think that needs to be in
> the release notes for the repair tool if that behavior remains when it
> ships.
>
> B.
>
> On Tue, Aug 10, 2010 at 9:09 AM, Mikeal Rogers <mi...@gmail.com> wrote:
>> I think I found a bug in the current lost+found repair.
>>
>> I've been running it against the testwritesdb and it's in a state that is
>> never finishing.
>>
>> It's still spitting out these lines:
>>
>> [info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
>>
>> Most are 1001 but there are also other random variances 452, 866, etc.
>>
>> But the file size and dbinfo hasn't budged in over 30 minutes. The size is
>> stuck at 34300002 with the original db file being 54857478 .
>>
>> This database only has one document in it that isn't "lost" so if it's
>> finding *any* new docs it should be writing them.
>>
>> I also started another job to recover a production db that is quite large,
>> 500megs, with the missing data a week or so back. This has been running for
>> 2 hours and has still not output anything or created the lost and found db
>> so I can only assume that it is in the same state.
>>
>> Both machines are still churning 100% CPU.
>>
>> -Mikeal
>>
>>
>> On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> With Randall's help we hooked the new node scanner up to the lost+found DB
>>> generator. It seems to work well enough for small DBs; for large DBs with
>>> lots of missing nodes the O(N^2) complexity of the problem catches up to the
>>> code and generating the lost+found DB takes quite some time. Mikeal is
>>> running tests tonight. The algo appears pretty CPU-limited, so a little
>>> parallelization may be warranted.
>>>
>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>
>>> Adam
>>>
>>> (I sent this previous update to myself instead of the list, so I'll forward
>>> it here ...)
>>>
>>> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
>>>
>>> > On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
>>> >
>>> >> Right, make_lost_and_found still relies on code which reads through
>>> couch_file one byte at a time, that's the cause of the slowness. The newer
>>> scanner will improve that pretty dramatically, and we can tune it further by
>>> increasing the length of the pattern that we match when looking for
>>> kp/kv_node terms in the files, at the expense of some extra complexity
>>> dealing with the block prefixes (currently it does a 1-byte match, which as
>>> I understand it cannot be split across blocks).
>>> >
>>> > The scanner now looks for a 7 byte match, unless it is within 6 bytes of
>>> a block boundary, in which case it looks for the longest possible match at
>>> that position. The more specific match condition greatly reduces the # of
>>> calls to couch_file, and thus boosts the throughput. On my laptop it can
>>> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
>>> seconds.
>>> >
>>> >> Regarding the file_corruption error on the larger file, I think this is
>>> something we will just naturally trigger when we take a guess that random
>>> positions in a file are actually the beginning of a term. I think our best
>>> recourse here is to return {error, file_corruption} from couch_file but
>>> leave the gen_server up and running instead of terminating it. That way the
>>> repair code can ignore the error and keep moving without having to reopen
>>> the file.
>>> >
>>> > I committed this change (to my db_repair branch) after consulting with
>>> Chris. The longer match condition makes these spurious file_corruption
>>> triggers much less likely, but I think it's still a good thing not to crash
>>> the server when they happen.
>>> >
>>> >> Next steps as I understand them - Randall is working on integrating the
>>> in-memory scanner into Volker's code that finds all the dangling by_id
>>> nodes. I'm working on making sure that the scanner identifies bt node
>>> candidates which span block prefixes, and on improving its pattern-matching.
>>> >
>>> > Latest from my end
>>> > http://github.com/kocolosk/couchdb/tree/db_repair
>>> >
>>> >>
>>> >> Adam
>>> >>
>>> >> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
>>> >>
>>> >>> I pulled down the latest code from Adam's branch @
>>> >>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>>> >>>
>>> >>> Running timer:tc(couch_db_repair, make_lost_and_found,
>>> ["multi_conflict"]).
>>> >>> on a database with 200 lost updates spanning 200 restarts (
>>> >>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch )
>>> took
>>> >>> about 101 seconds.
>>> >>>
>>> >>> I tried running against a larger databases (
>>> >>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch )
>>> and I
>>> >>> got this exception:
>>> >>>
>>> >>> http://gist.github.com/516491
>>> >>>
>>> >>> -Mikeal
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <randall.leeds@gmail.com
>>> >wrote:
>>> >>>
>>> >>>> Summing up what went on in IRC for those who were absent.
>>> >>>>
>>> >>>> The latest progress is on Adam's branch at
>>> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>> >>>>
>>> >>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>>> >>>> lost+found/DbName database to which it merges all nodes not accessible
>>> >>>> from anywhere (any other node found in a full file scan or any header
>>> >>>> pointers).
>>> >>>>
>>> >>>> Currently, make_lost_and_found uses Volker's repair (from
>>> >>>> couch_db_repair_b module, also in Adam's branch).
>>> >>>> Adam found that the bottleneck was couch_file calls and that the
>>> >>>> repair process was taking a very long time so he added
>>> >>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
>>> >>>> and tries to process it to find nodes instead of scanning back one
>>> >>>> byte at a time. It is currently not hooked up to the repair mechanism.
>>> >>>>
>>> >>>> Making progress. Go team.
>>> >>>>
>>> >>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
>>> >>>> wrote:
>>> >>>>> jchris suggested on IRC that I try a normal doc update and see if
>>> that
>>> >>>> fixes
>>> >>>>> it.
>>> >>>>>
>>> >>>>> It does. After a new doc was created the dbinfo doc count was back to
>>> >>>>> normal.
>>> >>>>>
>>> >>>>> -Mikeal
>>> >>>>>
>>> >>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
>>> mikeal.rogers@gmail.com
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> Ok, I pulled down this code and tested against a database with a ton
>>> of
>>> >>>>>> missing writes right before a single restart.
>>> >>>>>>
>>> >>>>>> Before restart this was the database:
>>> >>>>>>
>>> >>>>>> {
>>> >>>>>> db_name: "testwritesdb"
>>> >>>>>> doc_count: 124969
>>> >>>>>> doc_del_count: 0
>>> >>>>>> update_seq: 124969
>>> >>>>>> purge_seq: 0
>>> >>>>>> compact_running: false
>>> >>>>>> disk_size: 54857478
>>> >>>>>> instance_start_time: "1281384140058211"
>>> >>>>>> disk_format_version: 5
>>> >>>>>> }
>>> >>>>>>
>>> >>>>>> After restart it was this:
>>> >>>>>>
>>> >>>>>> {
>>> >>>>>> db_name: "testwritesdb"
>>> >>>>>> doc_count: 1
>>> >>>>>> doc_del_count: 0
>>> >>>>>> update_seq: 1
>>> >>>>>> purge_seq: 0
>>> >>>>>> compact_running: false
>>> >>>>>> disk_size: 54857478
>>> >>>>>> instance_start_time: "1281384593876026"
>>> >>>>>> disk_format_version: 5
>>> >>>>>> }
>>> >>>>>>
>>> >>>>>> After repair, it's this:
>>> >>>>>>
>>> >>>>>> {
>>> >>>>>> db_name: "testwritesdb"
>>> >>>>>> doc_count: 1
>>> >>>>>> doc_del_count: 0
>>> >>>>>> update_seq: 124969
>>> >>>>>> purge_seq: 0
>>> >>>>>> compact_running: false
>>> >>>>>> disk_size: 54857820
>>> >>>>>> instance_start_time: "1281385990193289"
>>> >>>>>> disk_format_version: 5
>>> >>>>>> committed_update_seq: 124969
>>> >>>>>> }
>>> >>>>>>
>>> >>>>>> All the sequences are there and hitting _all_docs shows all the
>>> >>>> documents
>>> >>>>>> so why is the doc_count only 1 in the dbinfo?
>>> >>>>>>
>>> >>>>>> -Mikeal
>>> >>>>>>
>>> >>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>>> >>>> fdmanana@apache.org>wrote:
>>> >>>>>>
>>> >>>>>>> For the record (and people not on IRC), the code at:
>>> >>>>>>>
>>> >>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>>> >>>>>>>
>>> >>>>>>> is working for at least simple cases. Use
>>> >>>>>>> couch_db_repair:repair(DbNameAsString).
>>> >>>>>>> There's one TODO: update the reduce values for the by_seq and
>>> by_id
>>> >>>>>>> BTrees.
>>> >>>>>>>
>>> >>>>>>> If anyone wants to give some help on this, your welcome.
>>> >>>>>>>
>>> >>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
>>> mikeal.rogers@gmail.com
>>> >>>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> I'm starting to create a bunch of test db files that expose this
>>> bug
>>> >>>>>>> under
>>> >>>>>>>> different conditions like multiple restarts, across compaction,
>>> >>>>>>> variances
>>> >>>>>>>> in
>>> >>>>>>>> updates the might cause conflict, etc.
>>> >>>>>>>>
>>> >>>>>>>> http://github.com/mikeal/couchtest
>>> >>>>>>>>
>>> >>>>>>>> The README outlines what was done to the db's and what needs to be
>>> >>>>>>>> recovered.
>>> >>>>>>>>
>>> >>>>>>>> -Mikeal
>>> >>>>>>>>
>>> >>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>>> >>>>>>> fdmanana@apache.org
>>> >>>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>>> >>>>>>> robert.newson@gmail.com
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Doesn't this bit;
>>> >>>>>>>>>>
>>> >>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>>> >>>>>>>>>> + Db;
>>> >>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>>> >>>>>>>>>>
>>> >>>>>>>>>> revert the bug fix?
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> That's intentional, for my local testing.
>>> >>>>>>>>> That patch isn't obviously anything close to final, it's too
>>> >>>>>>> experimental
>>> >>>>>>>>> yet.
>>> >>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> B.
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>>> >>>>>>> wrote:
>>> >>>>>>>>>>> Hi All,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but he
>>> >>>>>>> isn't
>>> >>>>>>>>> done
>>> >>>>>>>>>> yet.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Here's the current patch:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> it is not done and very early, but any help on this is greatly
>>> >>>>>>>>>> appreciated.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> The current state is (in Filipe's words):
>>> >>>>>>>>>>> - i can detect that a file needs repair
>>> >>>>>>>>>>> - and get the last btree roots from it
>>> >>>>>>>>>>> - "only" missing: get last db seq num
>>> >>>>>>>>>>> - write new header
>>> >>>>>>>>>>> - and deal with the local docs btree (if exists)
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Thanks!
>>> >>>>>>>>>>> Jan
>>> >>>>>>>>>>> --
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> --
>>> >>>>>>>>> Filipe David Manana,
>>> >>>>>>>>> fdmanana@apache.org
>>> >>>>>>>>>
>>> >>>>>>>>> "Reasonable men adapt themselves to the world.
>>> >>>>>>>>> Unreasonable men adapt the world to themselves.
>>> >>>>>>>>> That's why all progress depends on unreasonable men."
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Filipe David Manana,
>>> >>>>>>> fdmanana@apache.org
>>> >>>>>>>
>>> >>>>>>> "Reasonable men adapt themselves to the world.
>>> >>>>>>> Unreasonable men adapt the world to themselves.
>>> >>>>>>> That's why all progress depends on unreasonable men."
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> >
>>>
>>>
>>
>
Re: data recovery tool progress
Posted by Randall Leeds <ra...@gmail.com>.
I just pushed some more changes to my branch at
http://github.com/tilgovi/couchdb/tree/db_repair
At this point the algorithm is parallelized to configurable degree
(right now defined as ?WORKERS at the top).
On testwritesdb it seems to stop around 32M of written docs but the
process keeps working for a long time after (I didn't let it finish).
I suspect this is due to having found all (or most of all) of our
documents by descending from the roots we've already seen. It could
also be that Adam's term checking code doesn't quite catch all the
cases where terms are split over block boundary markers.
Sleep time.
Randall
Re: data recovery tool progress
Posted by Randall Leeds <ra...@gmail.com>.
Filipe,
I'm not sure which changes you're talking about exactly, but I know
Adam and I decided to use the old gen_server:call({pread that can read
arbitrary positions as binaries. The reason for this is so that the
scanner can read large chunks in one call and then analyze that for
node terms.
On Tue, Aug 10, 2010 at 02:46, Filipe David Manana <fd...@apache.org> wrote:
> Is it my impression or the forks I looked at (Volker, Adam, Randall) don't
> use the changes I made to couch_file? They were needed to try reading terms
> from random positions in the DB file, because if we try to read from a bad
> position, the couch_file gen_server crashed and was never restarted (it's
> not under a supervision tree).
>
>
> On Tue, Aug 10, 2010 at 10:28 AM, Filipe David Manana
> <fd...@apache.org>wrote:
>
>>
>>
>> On Tue, Aug 10, 2010 at 9:55 AM, Robert Newson <ro...@gmail.com>wrote:
>>
>>> In ran the db_repair code on a healthy database produced with
>>> delayed_commits=true.
>>>
>>> The source db had 3218 docs. db_repair recovered 3120 and then returned
>>> with ok.
>>>
>>
>> When a DB is repaired, couch_db_repair:repair/1 returns something matching
>> {ok, repaired, _BTreeInfos}.
>> If it returns only the atom 'ok' it means it did nothing to the DB file.
>> At least in my original code, dunno if the forks changed that behaviour.
>>
>>
>>>
>>> I'm redoing that test, but this indicates we're not finding all roots.
>>>
>>> I note that the output file was 36 times the input file, which is a
>>> consequence of folding all possible roots. I think that needs to be in
>>> the release notes for the repair tool if that behavior remains when it
>>> ships.
>>>
>>> B.
>>>
>>> On Tue, Aug 10, 2010 at 9:09 AM, Mikeal Rogers <mi...@gmail.com>
>>> wrote:
>>> > I think I found a bug in the current lost+found repair.
>>> >
>>> > I've been running it against the testwritesdb and it's in a state that
>>> is
>>> > never finishing.
>>> >
>>> > It's still spitting out these lines:
>>> >
>>> > [info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
>>> >
>>> > Most are 1001 but there are also other random variances 452, 866, etc.
>>> >
>>> > But the file size and dbinfo hasn't budged in over 30 minutes. The size
>>> is
>>> > stuck at 34300002 with the original db file being 54857478 .
>>> >
>>> > This database only has one document in it that isn't "lost" so if it's
>>> > finding *any* new docs it should be writing them.
>>> >
>>> > I also started another job to recover a production db that is quite
>>> large,
>>> > 500megs, with the missing data a week or so back. This has been running
>>> for
>>> > 2 hours and has still not output anything or created the lost and found
>>> db
>>> > so I can only assume that it is in the same state.
>>> >
>>> > Both machines are still churning 100% CPU.
>>> >
>>> > -Mikeal
>>> >
>>> >
>>> > On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org>
>>> wrote:
>>> >
>>> >> With Randall's help we hooked the new node scanner up to the lost+found
>>> DB
>>> >> generator. It seems to work well enough for small DBs; for large DBs
>>> with
>>> >> lots of missing nodes the O(N^2) complexity of the problem catches up
>>> to the
>>> >> code and generating the lost+found DB takes quite some time. Mikeal is
>>> >> running tests tonight. The algo appears pretty CPU-limited, so a
>>> little
>>> >> parallelization may be warranted.
>>> >>
>>> >> http://github.com/kocolosk/couchdb/tree/db_repair
>>> >>
>>> >> Adam
>>> >>
>>> >> (I sent this previous update to myself instead of the list, so I'll
>>> forward
>>> >> it here ...)
>>> >>
>>> >> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
>>> >>
>>> >> > On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
>>> >> >
>>> >> >> Right, make_lost_and_found still relies on code which reads through
>>> >> couch_file one byte at a time, that's the cause of the slowness. The
>>> newer
>>> >> scanner will improve that pretty dramatically, and we can tune it
>>> further by
>>> >> increasing the length of the pattern that we match when looking for
>>> >> kp/kv_node terms in the files, at the expense of some extra complexity
>>> >> dealing with the block prefixes (currently it does a 1-byte match,
>>> which as
>>> >> I understand it cannot be split across blocks).
>>> >> >
>>> >> > The scanner now looks for a 7 byte match, unless it is within 6 bytes
>>> of
>>> >> a block boundary, in which case it looks for the longest possible match
>>> at
>>> >> that position. The more specific match condition greatly reduces the #
>>> of
>>> >> calls to couch_file, and thus boosts the throughput. On my laptop it
>>> can
>>> >> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
>>> >> seconds.
>>> >> >
>>> >> >> Regarding the file_corruption error on the larger file, I think this
>>> is
>>> >> something we will just naturally trigger when we take a guess that
>>> random
>>> >> positions in a file are actually the beginning of a term. I think our
>>> best
>>> >> recourse here is to return {error, file_corruption} from couch_file but
>>> >> leave the gen_server up and running instead of terminating it. That
>>> way the
>>> >> repair code can ignore the error and keep moving without having to
>>> reopen
>>> >> the file.
>>> >> >
>>> >> > I committed this change (to my db_repair branch) after consulting
>>> with
>>> >> Chris. The longer match condition makes these spurious file_corruption
>>> >> triggers much less likely, but I think it's still a good thing not to
>>> crash
>>> >> the server when they happen.
>>> >> >
>>> >> >> Next steps as I understand them - Randall is working on integrating
>>> the
>>> >> in-memory scanner into Volker's code that finds all the dangling by_id
>>> >> nodes. I'm working on making sure that the scanner identifies bt node
>>> >> candidates which span block prefixes, and on improving its
>>> pattern-matching.
>>> >> >
>>> >> > Latest from my end
>>> >> > http://github.com/kocolosk/couchdb/tree/db_repair
>>> >> >
>>> >> >>
>>> >> >> Adam
>>> >> >>
>>> >> >> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
>>> >> >>
>>> >> >>> I pulled down the latest code from Adam's branch @
>>> >> >>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>>> >> >>>
>>> >> >>> Running timer:tc(couch_db_repair, make_lost_and_found,
>>> >> ["multi_conflict"]).
>>> >> >>> on a database with 200 lost updates spanning 200 restarts (
>>> >> >>>
>>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch )
>>> >> took
>>> >> >>> about 101 seconds.
>>> >> >>>
>>> >> >>> I tried running against a larger databases (
>>> >> >>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch)
>>> >> and I
>>> >> >>> got this exception:
>>> >> >>>
>>> >> >>> http://gist.github.com/516491
>>> >> >>>
>>> >> >>> -Mikeal
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <
>>> randall.leeds@gmail.com
>>> >> >wrote:
>>> >> >>>
>>> >> >>>> Summing up what went on in IRC for those who were absent.
>>> >> >>>>
>>> >> >>>> The latest progress is on Adam's branch at
>>> >> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>> >> >>>>
>>> >> >>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>>> >> >>>> lost+found/DbName database to which it merges all nodes not
>>> accessible
>>> >> >>>> from anywhere (any other node found in a full file scan or any
>>> header
>>> >> >>>> pointers).
>>> >> >>>>
>>> >> >>>> Currently, make_lost_and_found uses Volker's repair (from
>>> >> >>>> couch_db_repair_b module, also in Adam's branch).
>>> >> >>>> Adam found that the bottleneck was couch_file calls and that the
>>> >> >>>> repair process was taking a very long time so he added
>>> >> >>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as
>>> binary
>>> >> >>>> and tries to process it to find nodes instead of scanning back one
>>> >> >>>> byte at a time. It is currently not hooked up to the repair
>>> mechanism.
>>> >> >>>>
>>> >> >>>> Making progress. Go team.
>>> >> >>>>
>>> >> >>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <
>>> mikeal.rogers@gmail.com>
>>> >> >>>> wrote:
>>> >> >>>>> jchris suggested on IRC that I try a normal doc update and see if
>>> >> that
>>> >> >>>> fixes
>>> >> >>>>> it.
>>> >> >>>>>
>>> >> >>>>> It does. After a new doc was created the dbinfo doc count was
>>> back to
>>> >> >>>>> normal.
>>> >> >>>>>
>>> >> >>>>> -Mikeal
>>> >> >>>>>
>>> >> >>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
>>> >> mikeal.rogers@gmail.com
>>> >> >>>>> wrote:
>>> >> >>>>>
>>> >> >>>>>> Ok, I pulled down this code and tested against a database with a
>>> ton
>>> >> of
>>> >> >>>>>> missing writes right before a single restart.
>>> >> >>>>>>
>>> >> >>>>>> Before restart this was the database:
>>> >> >>>>>>
>>> >> >>>>>> {
>>> >> >>>>>> db_name: "testwritesdb"
>>> >> >>>>>> doc_count: 124969
>>> >> >>>>>> doc_del_count: 0
>>> >> >>>>>> update_seq: 124969
>>> >> >>>>>> purge_seq: 0
>>> >> >>>>>> compact_running: false
>>> >> >>>>>> disk_size: 54857478
>>> >> >>>>>> instance_start_time: "1281384140058211"
>>> >> >>>>>> disk_format_version: 5
>>> >> >>>>>> }
>>> >> >>>>>>
>>> >> >>>>>> After restart it was this:
>>> >> >>>>>>
>>> >> >>>>>> {
>>> >> >>>>>> db_name: "testwritesdb"
>>> >> >>>>>> doc_count: 1
>>> >> >>>>>> doc_del_count: 0
>>> >> >>>>>> update_seq: 1
>>> >> >>>>>> purge_seq: 0
>>> >> >>>>>> compact_running: false
>>> >> >>>>>> disk_size: 54857478
>>> >> >>>>>> instance_start_time: "1281384593876026"
>>> >> >>>>>> disk_format_version: 5
>>> >> >>>>>> }
>>> >> >>>>>>
>>> >> >>>>>> After repair, it's this:
>>> >> >>>>>>
>>> >> >>>>>> {
>>> >> >>>>>> db_name: "testwritesdb"
>>> >> >>>>>> doc_count: 1
>>> >> >>>>>> doc_del_count: 0
>>> >> >>>>>> update_seq: 124969
>>> >> >>>>>> purge_seq: 0
>>> >> >>>>>> compact_running: false
>>> >> >>>>>> disk_size: 54857820
>>> >> >>>>>> instance_start_time: "1281385990193289"
>>> >> >>>>>> disk_format_version: 5
>>> >> >>>>>> committed_update_seq: 124969
>>> >> >>>>>> }
>>> >> >>>>>>
>>> >> >>>>>> All the sequences are there and hitting _all_docs shows all the
>>> >> >>>> documents
>>> >> >>>>>> so why is the doc_count only 1 in the dbinfo?
>>> >> >>>>>>
>>> >> >>>>>> -Mikeal
>>> >> >>>>>>
>>> >> >>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>>> >> >>>> fdmanana@apache.org>wrote:
>>> >> >>>>>>
>>> >> >>>>>>> For the record (and people not on IRC), the code at:
>>> >> >>>>>>>
>>> >> >>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>>> >> >>>>>>>
>>> >> >>>>>>> is working for at least simple cases. Use
>>> >> >>>>>>> couch_db_repair:repair(DbNameAsString).
>>> >> >>>>>>> There's one TODO: update the reduce values for the by_seq and
>>> >> by_id
>>> >> >>>>>>> BTrees.
>>> >> >>>>>>>
>>> >> >>>>>>> If anyone wants to give some help on this, your welcome.
>>> >> >>>>>>>
>>> >> >>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
>>> >> mikeal.rogers@gmail.com
>>> >> >>>>>>>> wrote:
>>> >> >>>>>>>
>>> >> >>>>>>>> I'm starting to create a bunch of test db files that expose
>>> this
>>> >> bug
>>> >> >>>>>>> under
>>> >> >>>>>>>> different conditions like multiple restarts, across
>>> compaction,
>>> >> >>>>>>> variances
>>> >> >>>>>>>> in
>>> >> >>>>>>>> updates the might cause conflict, etc.
>>> >> >>>>>>>>
>>> >> >>>>>>>> http://github.com/mikeal/couchtest
>>> >> >>>>>>>>
>>> >> >>>>>>>> The README outlines what was done to the db's and what needs
>>> to be
>>> >> >>>>>>>> recovered.
>>> >> >>>>>>>>
>>> >> >>>>>>>> -Mikeal
>>> >> >>>>>>>>
>>> >> >>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>>> >> >>>>>>> fdmanana@apache.org
>>> >> >>>>>>>>> wrote:
>>> >> >>>>>>>>
>>> >> >>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>>> >> >>>>>>> robert.newson@gmail.com
>>> >> >>>>>>>>>> wrote:
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>> Doesn't this bit;
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>>> >> >>>>>>>>>> + Db;
>>> >> >>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> revert the bug fix?
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> That's intentional, for my local testing.
>>> >> >>>>>>>>> That patch isn't obviously anything close to final, it's too
>>> >> >>>>>>> experimental
>>> >> >>>>>>>>> yet.
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> B.
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <
>>> jan@apache.org>
>>> >> >>>>>>> wrote:
>>> >> >>>>>>>>>>> Hi All,
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but
>>> he
>>> >> >>>>>>> isn't
>>> >> >>>>>>>>> done
>>> >> >>>>>>>>>> yet.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Here's the current patch:
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> it is not done and very early, but any help on this is
>>> greatly
>>> >> >>>>>>>>>> appreciated.
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> The current state is (in Filipe's words):
>>> >> >>>>>>>>>>> - i can detect that a file needs repair
>>> >> >>>>>>>>>>> - and get the last btree roots from it
>>> >> >>>>>>>>>>> - "only" missing: get last db seq num
>>> >> >>>>>>>>>>> - write new header
>>> >> >>>>>>>>>>> - and deal with the local docs btree (if exists)
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>> Thanks!
>>> >> >>>>>>>>>>> Jan
>>> >> >>>>>>>>>>> --
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>>
>>> >> >>>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> --
>>> >> >>>>>>>>> Filipe David Manana,
>>> >> >>>>>>>>> fdmanana@apache.org
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> "Reasonable men adapt themselves to the world.
>>> >> >>>>>>>>> Unreasonable men adapt the world to themselves.
>>> >> >>>>>>>>> That's why all progress depends on unreasonable men."
>>> >> >>>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> --
>>> >> >>>>>>> Filipe David Manana,
>>> >> >>>>>>> fdmanana@apache.org
>>> >> >>>>>>>
>>> >> >>>>>>> "Reasonable men adapt themselves to the world.
>>> >> >>>>>>> Unreasonable men adapt the world to themselves.
>>> >> >>>>>>> That's why all progress depends on unreasonable men."
>>> >> >>>>>>>
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>
>>> >> >>>>
>>> >> >>
>>> >> >
>>> >>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana@apache.org
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>
>
> --
> Filipe David Manana,
> fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
Re: data recovery tool progress
Posted by Filipe David Manana <fd...@apache.org>.
Is it my impression or the forks I looked at (Volker, Adam, Randall) don't
use the changes I made to couch_file? They were needed to try reading terms
from random positions in the DB file, because if we try to read from a bad
position, the couch_file gen_server crashed and was never restarted (it's
not under a supervision tree).
On Tue, Aug 10, 2010 at 10:28 AM, Filipe David Manana
<fd...@apache.org>wrote:
>
>
> On Tue, Aug 10, 2010 at 9:55 AM, Robert Newson <ro...@gmail.com>wrote:
>
>> In ran the db_repair code on a healthy database produced with
>> delayed_commits=true.
>>
>> The source db had 3218 docs. db_repair recovered 3120 and then returned
>> with ok.
>>
>
> When a DB is repaired, couch_db_repair:repair/1 returns something matching
> {ok, repaired, _BTreeInfos}.
> If it returns only the atom 'ok' it means it did nothing to the DB file.
> At least in my original code, dunno if the forks changed that behaviour.
>
>
>>
>> I'm redoing that test, but this indicates we're not finding all roots.
>>
>> I note that the output file was 36 times the input file, which is a
>> consequence of folding all possible roots. I think that needs to be in
>> the release notes for the repair tool if that behavior remains when it
>> ships.
>>
>> B.
>>
>> On Tue, Aug 10, 2010 at 9:09 AM, Mikeal Rogers <mi...@gmail.com>
>> wrote:
>> > I think I found a bug in the current lost+found repair.
>> >
>> > I've been running it against the testwritesdb and it's in a state that
>> is
>> > never finishing.
>> >
>> > It's still spitting out these lines:
>> >
>> > [info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
>> >
>> > Most are 1001 but there are also other random variances 452, 866, etc.
>> >
>> > But the file size and dbinfo hasn't budged in over 30 minutes. The size
>> is
>> > stuck at 34300002 with the original db file being 54857478 .
>> >
>> > This database only has one document in it that isn't "lost" so if it's
>> > finding *any* new docs it should be writing them.
>> >
>> > I also started another job to recover a production db that is quite
>> large,
>> > 500megs, with the missing data a week or so back. This has been running
>> for
>> > 2 hours and has still not output anything or created the lost and found
>> db
>> > so I can only assume that it is in the same state.
>> >
>> > Both machines are still churning 100% CPU.
>> >
>> > -Mikeal
>> >
>> >
>> > On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org>
>> wrote:
>> >
>> >> With Randall's help we hooked the new node scanner up to the lost+found
>> DB
>> >> generator. It seems to work well enough for small DBs; for large DBs
>> with
>> >> lots of missing nodes the O(N^2) complexity of the problem catches up
>> to the
>> >> code and generating the lost+found DB takes quite some time. Mikeal is
>> >> running tests tonight. The algo appears pretty CPU-limited, so a
>> little
>> >> parallelization may be warranted.
>> >>
>> >> http://github.com/kocolosk/couchdb/tree/db_repair
>> >>
>> >> Adam
>> >>
>> >> (I sent this previous update to myself instead of the list, so I'll
>> forward
>> >> it here ...)
>> >>
>> >> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
>> >>
>> >> > On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
>> >> >
>> >> >> Right, make_lost_and_found still relies on code which reads through
>> >> couch_file one byte at a time, that's the cause of the slowness. The
>> newer
>> >> scanner will improve that pretty dramatically, and we can tune it
>> further by
>> >> increasing the length of the pattern that we match when looking for
>> >> kp/kv_node terms in the files, at the expense of some extra complexity
>> >> dealing with the block prefixes (currently it does a 1-byte match,
>> which as
>> >> I understand it cannot be split across blocks).
>> >> >
>> >> > The scanner now looks for a 7 byte match, unless it is within 6 bytes
>> of
>> >> a block boundary, in which case it looks for the longest possible match
>> at
>> >> that position. The more specific match condition greatly reduces the #
>> of
>> >> calls to couch_file, and thus boosts the throughput. On my laptop it
>> can
>> >> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
>> >> seconds.
>> >> >
>> >> >> Regarding the file_corruption error on the larger file, I think this
>> is
>> >> something we will just naturally trigger when we take a guess that
>> random
>> >> positions in a file are actually the beginning of a term. I think our
>> best
>> >> recourse here is to return {error, file_corruption} from couch_file but
>> >> leave the gen_server up and running instead of terminating it. That
>> way the
>> >> repair code can ignore the error and keep moving without having to
>> reopen
>> >> the file.
>> >> >
>> >> > I committed this change (to my db_repair branch) after consulting
>> with
>> >> Chris. The longer match condition makes these spurious file_corruption
>> >> triggers much less likely, but I think it's still a good thing not to
>> crash
>> >> the server when they happen.
>> >> >
>> >> >> Next steps as I understand them - Randall is working on integrating
>> the
>> >> in-memory scanner into Volker's code that finds all the dangling by_id
>> >> nodes. I'm working on making sure that the scanner identifies bt node
>> >> candidates which span block prefixes, and on improving its
>> pattern-matching.
>> >> >
>> >> > Latest from my end
>> >> > http://github.com/kocolosk/couchdb/tree/db_repair
>> >> >
>> >> >>
>> >> >> Adam
>> >> >>
>> >> >> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
>> >> >>
>> >> >>> I pulled down the latest code from Adam's branch @
>> >> >>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>> >> >>>
>> >> >>> Running timer:tc(couch_db_repair, make_lost_and_found,
>> >> ["multi_conflict"]).
>> >> >>> on a database with 200 lost updates spanning 200 restarts (
>> >> >>>
>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch )
>> >> took
>> >> >>> about 101 seconds.
>> >> >>>
>> >> >>> I tried running against a larger databases (
>> >> >>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch)
>> >> and I
>> >> >>> got this exception:
>> >> >>>
>> >> >>> http://gist.github.com/516491
>> >> >>>
>> >> >>> -Mikeal
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <
>> randall.leeds@gmail.com
>> >> >wrote:
>> >> >>>
>> >> >>>> Summing up what went on in IRC for those who were absent.
>> >> >>>>
>> >> >>>> The latest progress is on Adam's branch at
>> >> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
>> >> >>>>
>> >> >>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>> >> >>>> lost+found/DbName database to which it merges all nodes not
>> accessible
>> >> >>>> from anywhere (any other node found in a full file scan or any
>> header
>> >> >>>> pointers).
>> >> >>>>
>> >> >>>> Currently, make_lost_and_found uses Volker's repair (from
>> >> >>>> couch_db_repair_b module, also in Adam's branch).
>> >> >>>> Adam found that the bottleneck was couch_file calls and that the
>> >> >>>> repair process was taking a very long time so he added
>> >> >>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as
>> binary
>> >> >>>> and tries to process it to find nodes instead of scanning back one
>> >> >>>> byte at a time. It is currently not hooked up to the repair
>> mechanism.
>> >> >>>>
>> >> >>>> Making progress. Go team.
>> >> >>>>
>> >> >>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <
>> mikeal.rogers@gmail.com>
>> >> >>>> wrote:
>> >> >>>>> jchris suggested on IRC that I try a normal doc update and see if
>> >> that
>> >> >>>> fixes
>> >> >>>>> it.
>> >> >>>>>
>> >> >>>>> It does. After a new doc was created the dbinfo doc count was
>> back to
>> >> >>>>> normal.
>> >> >>>>>
>> >> >>>>> -Mikeal
>> >> >>>>>
>> >> >>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
>> >> mikeal.rogers@gmail.com
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>> Ok, I pulled down this code and tested against a database with a
>> ton
>> >> of
>> >> >>>>>> missing writes right before a single restart.
>> >> >>>>>>
>> >> >>>>>> Before restart this was the database:
>> >> >>>>>>
>> >> >>>>>> {
>> >> >>>>>> db_name: "testwritesdb"
>> >> >>>>>> doc_count: 124969
>> >> >>>>>> doc_del_count: 0
>> >> >>>>>> update_seq: 124969
>> >> >>>>>> purge_seq: 0
>> >> >>>>>> compact_running: false
>> >> >>>>>> disk_size: 54857478
>> >> >>>>>> instance_start_time: "1281384140058211"
>> >> >>>>>> disk_format_version: 5
>> >> >>>>>> }
>> >> >>>>>>
>> >> >>>>>> After restart it was this:
>> >> >>>>>>
>> >> >>>>>> {
>> >> >>>>>> db_name: "testwritesdb"
>> >> >>>>>> doc_count: 1
>> >> >>>>>> doc_del_count: 0
>> >> >>>>>> update_seq: 1
>> >> >>>>>> purge_seq: 0
>> >> >>>>>> compact_running: false
>> >> >>>>>> disk_size: 54857478
>> >> >>>>>> instance_start_time: "1281384593876026"
>> >> >>>>>> disk_format_version: 5
>> >> >>>>>> }
>> >> >>>>>>
>> >> >>>>>> After repair, it's this:
>> >> >>>>>>
>> >> >>>>>> {
>> >> >>>>>> db_name: "testwritesdb"
>> >> >>>>>> doc_count: 1
>> >> >>>>>> doc_del_count: 0
>> >> >>>>>> update_seq: 124969
>> >> >>>>>> purge_seq: 0
>> >> >>>>>> compact_running: false
>> >> >>>>>> disk_size: 54857820
>> >> >>>>>> instance_start_time: "1281385990193289"
>> >> >>>>>> disk_format_version: 5
>> >> >>>>>> committed_update_seq: 124969
>> >> >>>>>> }
>> >> >>>>>>
>> >> >>>>>> All the sequences are there and hitting _all_docs shows all the
>> >> >>>> documents
>> >> >>>>>> so why is the doc_count only 1 in the dbinfo?
>> >> >>>>>>
>> >> >>>>>> -Mikeal
>> >> >>>>>>
>> >> >>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>> >> >>>> fdmanana@apache.org>wrote:
>> >> >>>>>>
>> >> >>>>>>> For the record (and people not on IRC), the code at:
>> >> >>>>>>>
>> >> >>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>> >> >>>>>>>
>> >> >>>>>>> is working for at least simple cases. Use
>> >> >>>>>>> couch_db_repair:repair(DbNameAsString).
>> >> >>>>>>> There's one TODO: update the reduce values for the by_seq and
>> >> by_id
>> >> >>>>>>> BTrees.
>> >> >>>>>>>
>> >> >>>>>>> If anyone wants to give some help on this, your welcome.
>> >> >>>>>>>
>> >> >>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
>> >> mikeal.rogers@gmail.com
>> >> >>>>>>>> wrote:
>> >> >>>>>>>
>> >> >>>>>>>> I'm starting to create a bunch of test db files that expose
>> this
>> >> bug
>> >> >>>>>>> under
>> >> >>>>>>>> different conditions like multiple restarts, across
>> compaction,
>> >> >>>>>>> variances
>> >> >>>>>>>> in
>> >> >>>>>>>> updates the might cause conflict, etc.
>> >> >>>>>>>>
>> >> >>>>>>>> http://github.com/mikeal/couchtest
>> >> >>>>>>>>
>> >> >>>>>>>> The README outlines what was done to the db's and what needs
>> to be
>> >> >>>>>>>> recovered.
>> >> >>>>>>>>
>> >> >>>>>>>> -Mikeal
>> >> >>>>>>>>
>> >> >>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>> >> >>>>>>> fdmanana@apache.org
>> >> >>>>>>>>> wrote:
>> >> >>>>>>>>
>> >> >>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>> >> >>>>>>> robert.newson@gmail.com
>> >> >>>>>>>>>> wrote:
>> >> >>>>>>>>>
>> >> >>>>>>>>>> Doesn't this bit;
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>> >> >>>>>>>>>> + Db;
>> >> >>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> revert the bug fix?
>> >> >>>>>>>>>>
>> >> >>>>>>>>>
>> >> >>>>>>>>> That's intentional, for my local testing.
>> >> >>>>>>>>> That patch isn't obviously anything close to final, it's too
>> >> >>>>>>> experimental
>> >> >>>>>>>>> yet.
>> >> >>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> B.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <
>> jan@apache.org>
>> >> >>>>>>> wrote:
>> >> >>>>>>>>>>> Hi All,
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but
>> he
>> >> >>>>>>> isn't
>> >> >>>>>>>>> done
>> >> >>>>>>>>>> yet.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Here's the current patch:
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> it is not done and very early, but any help on this is
>> greatly
>> >> >>>>>>>>>> appreciated.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> The current state is (in Filipe's words):
>> >> >>>>>>>>>>> - i can detect that a file needs repair
>> >> >>>>>>>>>>> - and get the last btree roots from it
>> >> >>>>>>>>>>> - "only" missing: get last db seq num
>> >> >>>>>>>>>>> - write new header
>> >> >>>>>>>>>>> - and deal with the local docs btree (if exists)
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Thanks!
>> >> >>>>>>>>>>> Jan
>> >> >>>>>>>>>>> --
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>
>> >> >>>>>>>>>
>> >> >>>>>>>>>
>> >> >>>>>>>>> --
>> >> >>>>>>>>> Filipe David Manana,
>> >> >>>>>>>>> fdmanana@apache.org
>> >> >>>>>>>>>
>> >> >>>>>>>>> "Reasonable men adapt themselves to the world.
>> >> >>>>>>>>> Unreasonable men adapt the world to themselves.
>> >> >>>>>>>>> That's why all progress depends on unreasonable men."
>> >> >>>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> --
>> >> >>>>>>> Filipe David Manana,
>> >> >>>>>>> fdmanana@apache.org
>> >> >>>>>>>
>> >> >>>>>>> "Reasonable men adapt themselves to the world.
>> >> >>>>>>> Unreasonable men adapt the world to themselves.
>> >> >>>>>>> That's why all progress depends on unreasonable men."
>> >> >>>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>
>> >> >
>> >>
>> >>
>> >
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
--
Filipe David Manana,
fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: data recovery tool progress
Posted by Filipe David Manana <fd...@apache.org>.
On Tue, Aug 10, 2010 at 9:55 AM, Robert Newson <ro...@gmail.com>wrote:
> In ran the db_repair code on a healthy database produced with
> delayed_commits=true.
>
> The source db had 3218 docs. db_repair recovered 3120 and then returned
> with ok.
>
When a DB is repaired, couch_db_repair:repair/1 returns something matching
{ok, repaired, _BTreeInfos}.
If it returns only the atom 'ok' it means it did nothing to the DB file.
At least in my original code, dunno if the forks changed that behaviour.
>
> I'm redoing that test, but this indicates we're not finding all roots.
>
> I note that the output file was 36 times the input file, which is a
> consequence of folding all possible roots. I think that needs to be in
> the release notes for the repair tool if that behavior remains when it
> ships.
>
> B.
>
> On Tue, Aug 10, 2010 at 9:09 AM, Mikeal Rogers <mi...@gmail.com>
> wrote:
> > I think I found a bug in the current lost+found repair.
> >
> > I've been running it against the testwritesdb and it's in a state that is
> > never finishing.
> >
> > It's still spitting out these lines:
> >
> > [info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
> >
> > Most are 1001 but there are also other random variances 452, 866, etc.
> >
> > But the file size and dbinfo hasn't budged in over 30 minutes. The size
> is
> > stuck at 34300002 with the original db file being 54857478 .
> >
> > This database only has one document in it that isn't "lost" so if it's
> > finding *any* new docs it should be writing them.
> >
> > I also started another job to recover a production db that is quite
> large,
> > 500megs, with the missing data a week or so back. This has been running
> for
> > 2 hours and has still not output anything or created the lost and found
> db
> > so I can only assume that it is in the same state.
> >
> > Both machines are still churning 100% CPU.
> >
> > -Mikeal
> >
> >
> > On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org>
> wrote:
> >
> >> With Randall's help we hooked the new node scanner up to the lost+found
> DB
> >> generator. It seems to work well enough for small DBs; for large DBs
> with
> >> lots of missing nodes the O(N^2) complexity of the problem catches up to
> the
> >> code and generating the lost+found DB takes quite some time. Mikeal is
> >> running tests tonight. The algo appears pretty CPU-limited, so a little
> >> parallelization may be warranted.
> >>
> >> http://github.com/kocolosk/couchdb/tree/db_repair
> >>
> >> Adam
> >>
> >> (I sent this previous update to myself instead of the list, so I'll
> forward
> >> it here ...)
> >>
> >> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
> >>
> >> > On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
> >> >
> >> >> Right, make_lost_and_found still relies on code which reads through
> >> couch_file one byte at a time, that's the cause of the slowness. The
> newer
> >> scanner will improve that pretty dramatically, and we can tune it
> further by
> >> increasing the length of the pattern that we match when looking for
> >> kp/kv_node terms in the files, at the expense of some extra complexity
> >> dealing with the block prefixes (currently it does a 1-byte match, which
> as
> >> I understand it cannot be split across blocks).
> >> >
> >> > The scanner now looks for a 7 byte match, unless it is within 6 bytes
> of
> >> a block boundary, in which case it looks for the longest possible match
> at
> >> that position. The more specific match condition greatly reduces the #
> of
> >> calls to couch_file, and thus boosts the throughput. On my laptop it
> can
> >> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
> >> seconds.
> >> >
> >> >> Regarding the file_corruption error on the larger file, I think this
> is
> >> something we will just naturally trigger when we take a guess that
> random
> >> positions in a file are actually the beginning of a term. I think our
> best
> >> recourse here is to return {error, file_corruption} from couch_file but
> >> leave the gen_server up and running instead of terminating it. That way
> the
> >> repair code can ignore the error and keep moving without having to
> reopen
> >> the file.
> >> >
> >> > I committed this change (to my db_repair branch) after consulting with
> >> Chris. The longer match condition makes these spurious file_corruption
> >> triggers much less likely, but I think it's still a good thing not to
> crash
> >> the server when they happen.
> >> >
> >> >> Next steps as I understand them - Randall is working on integrating
> the
> >> in-memory scanner into Volker's code that finds all the dangling by_id
> >> nodes. I'm working on making sure that the scanner identifies bt node
> >> candidates which span block prefixes, and on improving its
> pattern-matching.
> >> >
> >> > Latest from my end
> >> > http://github.com/kocolosk/couchdb/tree/db_repair
> >> >
> >> >>
> >> >> Adam
> >> >>
> >> >> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
> >> >>
> >> >>> I pulled down the latest code from Adam's branch @
> >> >>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
> >> >>>
> >> >>> Running timer:tc(couch_db_repair, make_lost_and_found,
> >> ["multi_conflict"]).
> >> >>> on a database with 200 lost updates spanning 200 restarts (
> >> >>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch)
> >> took
> >> >>> about 101 seconds.
> >> >>>
> >> >>> I tried running against a larger databases (
> >> >>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch )
> >> and I
> >> >>> got this exception:
> >> >>>
> >> >>> http://gist.github.com/516491
> >> >>>
> >> >>> -Mikeal
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <
> randall.leeds@gmail.com
> >> >wrote:
> >> >>>
> >> >>>> Summing up what went on in IRC for those who were absent.
> >> >>>>
> >> >>>> The latest progress is on Adam's branch at
> >> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
> >> >>>>
> >> >>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
> >> >>>> lost+found/DbName database to which it merges all nodes not
> accessible
> >> >>>> from anywhere (any other node found in a full file scan or any
> header
> >> >>>> pointers).
> >> >>>>
> >> >>>> Currently, make_lost_and_found uses Volker's repair (from
> >> >>>> couch_db_repair_b module, also in Adam's branch).
> >> >>>> Adam found that the bottleneck was couch_file calls and that the
> >> >>>> repair process was taking a very long time so he added
> >> >>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as
> binary
> >> >>>> and tries to process it to find nodes instead of scanning back one
> >> >>>> byte at a time. It is currently not hooked up to the repair
> mechanism.
> >> >>>>
> >> >>>> Making progress. Go team.
> >> >>>>
> >> >>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <
> mikeal.rogers@gmail.com>
> >> >>>> wrote:
> >> >>>>> jchris suggested on IRC that I try a normal doc update and see if
> >> that
> >> >>>> fixes
> >> >>>>> it.
> >> >>>>>
> >> >>>>> It does. After a new doc was created the dbinfo doc count was back
> to
> >> >>>>> normal.
> >> >>>>>
> >> >>>>> -Mikeal
> >> >>>>>
> >> >>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
> >> mikeal.rogers@gmail.com
> >> >>>>> wrote:
> >> >>>>>
> >> >>>>>> Ok, I pulled down this code and tested against a database with a
> ton
> >> of
> >> >>>>>> missing writes right before a single restart.
> >> >>>>>>
> >> >>>>>> Before restart this was the database:
> >> >>>>>>
> >> >>>>>> {
> >> >>>>>> db_name: "testwritesdb"
> >> >>>>>> doc_count: 124969
> >> >>>>>> doc_del_count: 0
> >> >>>>>> update_seq: 124969
> >> >>>>>> purge_seq: 0
> >> >>>>>> compact_running: false
> >> >>>>>> disk_size: 54857478
> >> >>>>>> instance_start_time: "1281384140058211"
> >> >>>>>> disk_format_version: 5
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> After restart it was this:
> >> >>>>>>
> >> >>>>>> {
> >> >>>>>> db_name: "testwritesdb"
> >> >>>>>> doc_count: 1
> >> >>>>>> doc_del_count: 0
> >> >>>>>> update_seq: 1
> >> >>>>>> purge_seq: 0
> >> >>>>>> compact_running: false
> >> >>>>>> disk_size: 54857478
> >> >>>>>> instance_start_time: "1281384593876026"
> >> >>>>>> disk_format_version: 5
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> After repair, it's this:
> >> >>>>>>
> >> >>>>>> {
> >> >>>>>> db_name: "testwritesdb"
> >> >>>>>> doc_count: 1
> >> >>>>>> doc_del_count: 0
> >> >>>>>> update_seq: 124969
> >> >>>>>> purge_seq: 0
> >> >>>>>> compact_running: false
> >> >>>>>> disk_size: 54857820
> >> >>>>>> instance_start_time: "1281385990193289"
> >> >>>>>> disk_format_version: 5
> >> >>>>>> committed_update_seq: 124969
> >> >>>>>> }
> >> >>>>>>
> >> >>>>>> All the sequences are there and hitting _all_docs shows all the
> >> >>>> documents
> >> >>>>>> so why is the doc_count only 1 in the dbinfo?
> >> >>>>>>
> >> >>>>>> -Mikeal
> >> >>>>>>
> >> >>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
> >> >>>> fdmanana@apache.org>wrote:
> >> >>>>>>
> >> >>>>>>> For the record (and people not on IRC), the code at:
> >> >>>>>>>
> >> >>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
> >> >>>>>>>
> >> >>>>>>> is working for at least simple cases. Use
> >> >>>>>>> couch_db_repair:repair(DbNameAsString).
> >> >>>>>>> There's one TODO: update the reduce values for the by_seq and
> >> by_id
> >> >>>>>>> BTrees.
> >> >>>>>>>
> >> >>>>>>> If anyone wants to give some help on this, your welcome.
> >> >>>>>>>
> >> >>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
> >> mikeal.rogers@gmail.com
> >> >>>>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>>> I'm starting to create a bunch of test db files that expose
> this
> >> bug
> >> >>>>>>> under
> >> >>>>>>>> different conditions like multiple restarts, across compaction,
> >> >>>>>>> variances
> >> >>>>>>>> in
> >> >>>>>>>> updates the might cause conflict, etc.
> >> >>>>>>>>
> >> >>>>>>>> http://github.com/mikeal/couchtest
> >> >>>>>>>>
> >> >>>>>>>> The README outlines what was done to the db's and what needs to
> be
> >> >>>>>>>> recovered.
> >> >>>>>>>>
> >> >>>>>>>> -Mikeal
> >> >>>>>>>>
> >> >>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
> >> >>>>>>> fdmanana@apache.org
> >> >>>>>>>>> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
> >> >>>>>>> robert.newson@gmail.com
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>> Doesn't this bit;
> >> >>>>>>>>>>
> >> >>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
> >> >>>>>>>>>> + Db;
> >> >>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
> >> >>>>>>>>>>
> >> >>>>>>>>>> revert the bug fix?
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> That's intentional, for my local testing.
> >> >>>>>>>>> That patch isn't obviously anything close to final, it's too
> >> >>>>>>> experimental
> >> >>>>>>>>> yet.
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> B.
> >> >>>>>>>>>>
> >> >>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <jan@apache.org
> >
> >> >>>>>>> wrote:
> >> >>>>>>>>>>> Hi All,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but
> he
> >> >>>>>>> isn't
> >> >>>>>>>>> done
> >> >>>>>>>>>> yet.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Here's the current patch:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> it is not done and very early, but any help on this is
> greatly
> >> >>>>>>>>>> appreciated.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> The current state is (in Filipe's words):
> >> >>>>>>>>>>> - i can detect that a file needs repair
> >> >>>>>>>>>>> - and get the last btree roots from it
> >> >>>>>>>>>>> - "only" missing: get last db seq num
> >> >>>>>>>>>>> - write new header
> >> >>>>>>>>>>> - and deal with the local docs btree (if exists)
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Thanks!
> >> >>>>>>>>>>> Jan
> >> >>>>>>>>>>> --
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> --
> >> >>>>>>>>> Filipe David Manana,
> >> >>>>>>>>> fdmanana@apache.org
> >> >>>>>>>>>
> >> >>>>>>>>> "Reasonable men adapt themselves to the world.
> >> >>>>>>>>> Unreasonable men adapt the world to themselves.
> >> >>>>>>>>> That's why all progress depends on unreasonable men."
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> --
> >> >>>>>>> Filipe David Manana,
> >> >>>>>>> fdmanana@apache.org
> >> >>>>>>>
> >> >>>>>>> "Reasonable men adapt themselves to the world.
> >> >>>>>>> Unreasonable men adapt the world to themselves.
> >> >>>>>>> That's why all progress depends on unreasonable men."
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>
> >> >
> >>
> >>
> >
>
--
Filipe David Manana,
fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
On Aug 10, 2010, at 6:27 AM, Jan Lehnardt wrote:
> This looks like we are recovering nodes that don't need recovering because on a healthy db produced with delayed_commits=true we should not have any orphans at all, so the lost and found db should be empty.
Yes, that would be ideal. I think the issue is that "prune_nodes" ends up returning all btree roots except the latest one. It doesn't actually grab all the headers and remove root nodes they point to, it only does this for the latest header. As a result, I think rnewson's results make sense.
On Aug 10, 2010, at 4:55 AM, Robert Newson wrote:
> In ran the db_repair code on a healthy database produced with
> delayed_commits=true.
>
> The source db had 3218 docs. db_repair recovered 3120 and then returned with ok.
>
> I'm redoing that test, but this indicates we're not finding all roots.
On a healthy DB it should have found zero docs. It probably found all docs except those which are accessible only from the latest header.
On Aug 10, 2010, at 4:09 AM, Mikeal Rogers wrote:
> I think I found a bug in the current lost+found repair.
>
> I've been running it against the testwritesdb and it's in a state that is
> never finishing.
I'll bet it's working ok. prune_nodes() returns 26000+ roots for that DB, so we need to do 26000+ replications, each of which finds a large number of documents.
The solution is to prune more nodes.
Adam
Re: data recovery tool progress
Posted by Jan Lehnardt <ja...@apache.org>.
On 10 Aug 2010, at 10:55, Robert Newson wrote:
> In ran the db_repair code on a healthy database produced with
> delayed_commits=true.
>
> The source db had 3218 docs. db_repair recovered 3120 and then returned with ok.
This looks like we are recovering nodes that don't need recovering because on a healthy db produced with delayed_commits=true we should not have any orphans at all, so the lost and found db should be empty.
>
> I'm redoing that test, but this indicates we're not finding all roots.
>
> I note that the output file was 36 times the input file, which is a
> consequence of folding all possible roots. I think that needs to be in
> the release notes for the repair tool if that behavior remains when it
> ships.
>
> B.
>
> On Tue, Aug 10, 2010 at 9:09 AM, Mikeal Rogers <mi...@gmail.com> wrote:
>> I think I found a bug in the current lost+found repair.
>>
>> I've been running it against the testwritesdb and it's in a state that is
>> never finishing.
>>
>> It's still spitting out these lines:
>>
>> [info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
>>
>> Most are 1001 but there are also other random variances 452, 866, etc.
>>
>> But the file size and dbinfo hasn't budged in over 30 minutes. The size is
>> stuck at 34300002 with the original db file being 54857478 .
>>
>> This database only has one document in it that isn't "lost" so if it's
>> finding *any* new docs it should be writing them.
>>
>> I also started another job to recover a production db that is quite large,
>> 500megs, with the missing data a week or so back. This has been running for
>> 2 hours and has still not output anything or created the lost and found db
>> so I can only assume that it is in the same state.
>>
>> Both machines are still churning 100% CPU.
>>
>> -Mikeal
>>
>>
>> On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>
>>> With Randall's help we hooked the new node scanner up to the lost+found DB
>>> generator. It seems to work well enough for small DBs; for large DBs with
>>> lots of missing nodes the O(N^2) complexity of the problem catches up to the
>>> code and generating the lost+found DB takes quite some time. Mikeal is
>>> running tests tonight. The algo appears pretty CPU-limited, so a little
>>> parallelization may be warranted.
>>>
>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>
>>> Adam
>>>
>>> (I sent this previous update to myself instead of the list, so I'll forward
>>> it here ...)
>>>
>>> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
>>>
>>>> On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
>>>>
>>>>> Right, make_lost_and_found still relies on code which reads through
>>> couch_file one byte at a time, that's the cause of the slowness. The newer
>>> scanner will improve that pretty dramatically, and we can tune it further by
>>> increasing the length of the pattern that we match when looking for
>>> kp/kv_node terms in the files, at the expense of some extra complexity
>>> dealing with the block prefixes (currently it does a 1-byte match, which as
>>> I understand it cannot be split across blocks).
>>>>
>>>> The scanner now looks for a 7 byte match, unless it is within 6 bytes of
>>> a block boundary, in which case it looks for the longest possible match at
>>> that position. The more specific match condition greatly reduces the # of
>>> calls to couch_file, and thus boosts the throughput. On my laptop it can
>>> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
>>> seconds.
>>>>
>>>>> Regarding the file_corruption error on the larger file, I think this is
>>> something we will just naturally trigger when we take a guess that random
>>> positions in a file are actually the beginning of a term. I think our best
>>> recourse here is to return {error, file_corruption} from couch_file but
>>> leave the gen_server up and running instead of terminating it. That way the
>>> repair code can ignore the error and keep moving without having to reopen
>>> the file.
>>>>
>>>> I committed this change (to my db_repair branch) after consulting with
>>> Chris. The longer match condition makes these spurious file_corruption
>>> triggers much less likely, but I think it's still a good thing not to crash
>>> the server when they happen.
>>>>
>>>>> Next steps as I understand them - Randall is working on integrating the
>>> in-memory scanner into Volker's code that finds all the dangling by_id
>>> nodes. I'm working on making sure that the scanner identifies bt node
>>> candidates which span block prefixes, and on improving its pattern-matching.
>>>>
>>>> Latest from my end
>>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>>
>>>>>
>>>>> Adam
>>>>>
>>>>> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
>>>>>
>>>>>> I pulled down the latest code from Adam's branch @
>>>>>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>>>>>>
>>>>>> Running timer:tc(couch_db_repair, make_lost_and_found,
>>> ["multi_conflict"]).
>>>>>> on a database with 200 lost updates spanning 200 restarts (
>>>>>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch )
>>> took
>>>>>> about 101 seconds.
>>>>>>
>>>>>> I tried running against a larger databases (
>>>>>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch )
>>> and I
>>>>>> got this exception:
>>>>>>
>>>>>> http://gist.github.com/516491
>>>>>>
>>>>>> -Mikeal
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <randall.leeds@gmail.com
>>>> wrote:
>>>>>>
>>>>>>> Summing up what went on in IRC for those who were absent.
>>>>>>>
>>>>>>> The latest progress is on Adam's branch at
>>>>>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>>>>>
>>>>>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>>>>>>> lost+found/DbName database to which it merges all nodes not accessible
>>>>>>> from anywhere (any other node found in a full file scan or any header
>>>>>>> pointers).
>>>>>>>
>>>>>>> Currently, make_lost_and_found uses Volker's repair (from
>>>>>>> couch_db_repair_b module, also in Adam's branch).
>>>>>>> Adam found that the bottleneck was couch_file calls and that the
>>>>>>> repair process was taking a very long time so he added
>>>>>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
>>>>>>> and tries to process it to find nodes instead of scanning back one
>>>>>>> byte at a time. It is currently not hooked up to the repair mechanism.
>>>>>>>
>>>>>>> Making progress. Go team.
>>>>>>>
>>>>>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
>>>>>>> wrote:
>>>>>>>> jchris suggested on IRC that I try a normal doc update and see if
>>> that
>>>>>>> fixes
>>>>>>>> it.
>>>>>>>>
>>>>>>>> It does. After a new doc was created the dbinfo doc count was back to
>>>>>>>> normal.
>>>>>>>>
>>>>>>>> -Mikeal
>>>>>>>>
>>>>>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
>>> mikeal.rogers@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ok, I pulled down this code and tested against a database with a ton
>>> of
>>>>>>>>> missing writes right before a single restart.
>>>>>>>>>
>>>>>>>>> Before restart this was the database:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> db_name: "testwritesdb"
>>>>>>>>> doc_count: 124969
>>>>>>>>> doc_del_count: 0
>>>>>>>>> update_seq: 124969
>>>>>>>>> purge_seq: 0
>>>>>>>>> compact_running: false
>>>>>>>>> disk_size: 54857478
>>>>>>>>> instance_start_time: "1281384140058211"
>>>>>>>>> disk_format_version: 5
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> After restart it was this:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> db_name: "testwritesdb"
>>>>>>>>> doc_count: 1
>>>>>>>>> doc_del_count: 0
>>>>>>>>> update_seq: 1
>>>>>>>>> purge_seq: 0
>>>>>>>>> compact_running: false
>>>>>>>>> disk_size: 54857478
>>>>>>>>> instance_start_time: "1281384593876026"
>>>>>>>>> disk_format_version: 5
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> After repair, it's this:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> db_name: "testwritesdb"
>>>>>>>>> doc_count: 1
>>>>>>>>> doc_del_count: 0
>>>>>>>>> update_seq: 124969
>>>>>>>>> purge_seq: 0
>>>>>>>>> compact_running: false
>>>>>>>>> disk_size: 54857820
>>>>>>>>> instance_start_time: "1281385990193289"
>>>>>>>>> disk_format_version: 5
>>>>>>>>> committed_update_seq: 124969
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> All the sequences are there and hitting _all_docs shows all the
>>>>>>> documents
>>>>>>>>> so why is the doc_count only 1 in the dbinfo?
>>>>>>>>>
>>>>>>>>> -Mikeal
>>>>>>>>>
>>>>>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>>>>>>> fdmanana@apache.org>wrote:
>>>>>>>>>
>>>>>>>>>> For the record (and people not on IRC), the code at:
>>>>>>>>>>
>>>>>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>>>>>>>>>>
>>>>>>>>>> is working for at least simple cases. Use
>>>>>>>>>> couch_db_repair:repair(DbNameAsString).
>>>>>>>>>> There's one TODO: update the reduce values for the by_seq and
>>> by_id
>>>>>>>>>> BTrees.
>>>>>>>>>>
>>>>>>>>>> If anyone wants to give some help on this, your welcome.
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
>>> mikeal.rogers@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm starting to create a bunch of test db files that expose this
>>> bug
>>>>>>>>>> under
>>>>>>>>>>> different conditions like multiple restarts, across compaction,
>>>>>>>>>> variances
>>>>>>>>>>> in
>>>>>>>>>>> updates the might cause conflict, etc.
>>>>>>>>>>>
>>>>>>>>>>> http://github.com/mikeal/couchtest
>>>>>>>>>>>
>>>>>>>>>>> The README outlines what was done to the db's and what needs to be
>>>>>>>>>>> recovered.
>>>>>>>>>>>
>>>>>>>>>>> -Mikeal
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>>>>>>>>>> fdmanana@apache.org
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>>>>>>>>>> robert.newson@gmail.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Doesn't this bit;
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>>>>>>>>>>>>> + Db;
>>>>>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>>>>>>>>>>>>>
>>>>>>>>>>>>> revert the bug fix?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That's intentional, for my local testing.
>>>>>>>>>>>> That patch isn't obviously anything close to final, it's too
>>>>>>>>>> experimental
>>>>>>>>>>>> yet.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> B.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but he
>>>>>>>>>> isn't
>>>>>>>>>>>> done
>>>>>>>>>>>>> yet.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the current patch:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> it is not done and very early, but any help on this is greatly
>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The current state is (in Filipe's words):
>>>>>>>>>>>>>> - i can detect that a file needs repair
>>>>>>>>>>>>>> - and get the last btree roots from it
>>>>>>>>>>>>>> - "only" missing: get last db seq num
>>>>>>>>>>>>>> - write new header
>>>>>>>>>>>>>> - and deal with the local docs btree (if exists)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Filipe David Manana,
>>>>>>>>>>>> fdmanana@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>> "Reasonable men adapt themselves to the world.
>>>>>>>>>>>> Unreasonable men adapt the world to themselves.
>>>>>>>>>>>> That's why all progress depends on unreasonable men."
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Filipe David Manana,
>>>>>>>>>> fdmanana@apache.org
>>>>>>>>>>
>>>>>>>>>> "Reasonable men adapt themselves to the world.
>>>>>>>>>> Unreasonable men adapt the world to themselves.
>>>>>>>>>> That's why all progress depends on unreasonable men."
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>>
>>
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
In ran the db_repair code on a healthy database produced with
delayed_commits=true.
The source db had 3218 docs. db_repair recovered 3120 and then returned with ok.
I'm redoing that test, but this indicates we're not finding all roots.
I note that the output file was 36 times the input file, which is a
consequence of folding all possible roots. I think that needs to be in
the release notes for the repair tool if that behavior remains when it
ships.
B.
On Tue, Aug 10, 2010 at 9:09 AM, Mikeal Rogers <mi...@gmail.com> wrote:
> I think I found a bug in the current lost+found repair.
>
> I've been running it against the testwritesdb and it's in a state that is
> never finishing.
>
> It's still spitting out these lines:
>
> [info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
>
> Most are 1001 but there are also other random variances 452, 866, etc.
>
> But the file size and dbinfo hasn't budged in over 30 minutes. The size is
> stuck at 34300002 with the original db file being 54857478 .
>
> This database only has one document in it that isn't "lost" so if it's
> finding *any* new docs it should be writing them.
>
> I also started another job to recover a production db that is quite large,
> 500megs, with the missing data a week or so back. This has been running for
> 2 hours and has still not output anything or created the lost and found db
> so I can only assume that it is in the same state.
>
> Both machines are still churning 100% CPU.
>
> -Mikeal
>
>
> On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org> wrote:
>
>> With Randall's help we hooked the new node scanner up to the lost+found DB
>> generator. It seems to work well enough for small DBs; for large DBs with
>> lots of missing nodes the O(N^2) complexity of the problem catches up to the
>> code and generating the lost+found DB takes quite some time. Mikeal is
>> running tests tonight. The algo appears pretty CPU-limited, so a little
>> parallelization may be warranted.
>>
>> http://github.com/kocolosk/couchdb/tree/db_repair
>>
>> Adam
>>
>> (I sent this previous update to myself instead of the list, so I'll forward
>> it here ...)
>>
>> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
>>
>> > On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
>> >
>> >> Right, make_lost_and_found still relies on code which reads through
>> couch_file one byte at a time, that's the cause of the slowness. The newer
>> scanner will improve that pretty dramatically, and we can tune it further by
>> increasing the length of the pattern that we match when looking for
>> kp/kv_node terms in the files, at the expense of some extra complexity
>> dealing with the block prefixes (currently it does a 1-byte match, which as
>> I understand it cannot be split across blocks).
>> >
>> > The scanner now looks for a 7 byte match, unless it is within 6 bytes of
>> a block boundary, in which case it looks for the longest possible match at
>> that position. The more specific match condition greatly reduces the # of
>> calls to couch_file, and thus boosts the throughput. On my laptop it can
>> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
>> seconds.
>> >
>> >> Regarding the file_corruption error on the larger file, I think this is
>> something we will just naturally trigger when we take a guess that random
>> positions in a file are actually the beginning of a term. I think our best
>> recourse here is to return {error, file_corruption} from couch_file but
>> leave the gen_server up and running instead of terminating it. That way the
>> repair code can ignore the error and keep moving without having to reopen
>> the file.
>> >
>> > I committed this change (to my db_repair branch) after consulting with
>> Chris. The longer match condition makes these spurious file_corruption
>> triggers much less likely, but I think it's still a good thing not to crash
>> the server when they happen.
>> >
>> >> Next steps as I understand them - Randall is working on integrating the
>> in-memory scanner into Volker's code that finds all the dangling by_id
>> nodes. I'm working on making sure that the scanner identifies bt node
>> candidates which span block prefixes, and on improving its pattern-matching.
>> >
>> > Latest from my end
>> > http://github.com/kocolosk/couchdb/tree/db_repair
>> >
>> >>
>> >> Adam
>> >>
>> >> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
>> >>
>> >>> I pulled down the latest code from Adam's branch @
>> >>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>> >>>
>> >>> Running timer:tc(couch_db_repair, make_lost_and_found,
>> ["multi_conflict"]).
>> >>> on a database with 200 lost updates spanning 200 restarts (
>> >>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch )
>> took
>> >>> about 101 seconds.
>> >>>
>> >>> I tried running against a larger databases (
>> >>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch )
>> and I
>> >>> got this exception:
>> >>>
>> >>> http://gist.github.com/516491
>> >>>
>> >>> -Mikeal
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <randall.leeds@gmail.com
>> >wrote:
>> >>>
>> >>>> Summing up what went on in IRC for those who were absent.
>> >>>>
>> >>>> The latest progress is on Adam's branch at
>> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
>> >>>>
>> >>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>> >>>> lost+found/DbName database to which it merges all nodes not accessible
>> >>>> from anywhere (any other node found in a full file scan or any header
>> >>>> pointers).
>> >>>>
>> >>>> Currently, make_lost_and_found uses Volker's repair (from
>> >>>> couch_db_repair_b module, also in Adam's branch).
>> >>>> Adam found that the bottleneck was couch_file calls and that the
>> >>>> repair process was taking a very long time so he added
>> >>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
>> >>>> and tries to process it to find nodes instead of scanning back one
>> >>>> byte at a time. It is currently not hooked up to the repair mechanism.
>> >>>>
>> >>>> Making progress. Go team.
>> >>>>
>> >>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
>> >>>> wrote:
>> >>>>> jchris suggested on IRC that I try a normal doc update and see if
>> that
>> >>>> fixes
>> >>>>> it.
>> >>>>>
>> >>>>> It does. After a new doc was created the dbinfo doc count was back to
>> >>>>> normal.
>> >>>>>
>> >>>>> -Mikeal
>> >>>>>
>> >>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
>> mikeal.rogers@gmail.com
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Ok, I pulled down this code and tested against a database with a ton
>> of
>> >>>>>> missing writes right before a single restart.
>> >>>>>>
>> >>>>>> Before restart this was the database:
>> >>>>>>
>> >>>>>> {
>> >>>>>> db_name: "testwritesdb"
>> >>>>>> doc_count: 124969
>> >>>>>> doc_del_count: 0
>> >>>>>> update_seq: 124969
>> >>>>>> purge_seq: 0
>> >>>>>> compact_running: false
>> >>>>>> disk_size: 54857478
>> >>>>>> instance_start_time: "1281384140058211"
>> >>>>>> disk_format_version: 5
>> >>>>>> }
>> >>>>>>
>> >>>>>> After restart it was this:
>> >>>>>>
>> >>>>>> {
>> >>>>>> db_name: "testwritesdb"
>> >>>>>> doc_count: 1
>> >>>>>> doc_del_count: 0
>> >>>>>> update_seq: 1
>> >>>>>> purge_seq: 0
>> >>>>>> compact_running: false
>> >>>>>> disk_size: 54857478
>> >>>>>> instance_start_time: "1281384593876026"
>> >>>>>> disk_format_version: 5
>> >>>>>> }
>> >>>>>>
>> >>>>>> After repair, it's this:
>> >>>>>>
>> >>>>>> {
>> >>>>>> db_name: "testwritesdb"
>> >>>>>> doc_count: 1
>> >>>>>> doc_del_count: 0
>> >>>>>> update_seq: 124969
>> >>>>>> purge_seq: 0
>> >>>>>> compact_running: false
>> >>>>>> disk_size: 54857820
>> >>>>>> instance_start_time: "1281385990193289"
>> >>>>>> disk_format_version: 5
>> >>>>>> committed_update_seq: 124969
>> >>>>>> }
>> >>>>>>
>> >>>>>> All the sequences are there and hitting _all_docs shows all the
>> >>>> documents
>> >>>>>> so why is the doc_count only 1 in the dbinfo?
>> >>>>>>
>> >>>>>> -Mikeal
>> >>>>>>
>> >>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>> >>>> fdmanana@apache.org>wrote:
>> >>>>>>
>> >>>>>>> For the record (and people not on IRC), the code at:
>> >>>>>>>
>> >>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>> >>>>>>>
>> >>>>>>> is working for at least simple cases. Use
>> >>>>>>> couch_db_repair:repair(DbNameAsString).
>> >>>>>>> There's one TODO: update the reduce values for the by_seq and
>> by_id
>> >>>>>>> BTrees.
>> >>>>>>>
>> >>>>>>> If anyone wants to give some help on this, your welcome.
>> >>>>>>>
>> >>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
>> mikeal.rogers@gmail.com
>> >>>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> I'm starting to create a bunch of test db files that expose this
>> bug
>> >>>>>>> under
>> >>>>>>>> different conditions like multiple restarts, across compaction,
>> >>>>>>> variances
>> >>>>>>>> in
>> >>>>>>>> updates the might cause conflict, etc.
>> >>>>>>>>
>> >>>>>>>> http://github.com/mikeal/couchtest
>> >>>>>>>>
>> >>>>>>>> The README outlines what was done to the db's and what needs to be
>> >>>>>>>> recovered.
>> >>>>>>>>
>> >>>>>>>> -Mikeal
>> >>>>>>>>
>> >>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>> >>>>>>> fdmanana@apache.org
>> >>>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>> >>>>>>> robert.newson@gmail.com
>> >>>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Doesn't this bit;
>> >>>>>>>>>>
>> >>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>> >>>>>>>>>> + Db;
>> >>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>> >>>>>>>>>>
>> >>>>>>>>>> revert the bug fix?
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> That's intentional, for my local testing.
>> >>>>>>>>> That patch isn't obviously anything close to final, it's too
>> >>>>>>> experimental
>> >>>>>>>>> yet.
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> B.
>> >>>>>>>>>>
>> >>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>>>>> Hi All,
>> >>>>>>>>>>>
>> >>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but he
>> >>>>>>> isn't
>> >>>>>>>>> done
>> >>>>>>>>>> yet.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Here's the current patch:
>> >>>>>>>>>>>
>> >>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>> >>>>>>>>>>>
>> >>>>>>>>>>> it is not done and very early, but any help on this is greatly
>> >>>>>>>>>> appreciated.
>> >>>>>>>>>>>
>> >>>>>>>>>>> The current state is (in Filipe's words):
>> >>>>>>>>>>> - i can detect that a file needs repair
>> >>>>>>>>>>> - and get the last btree roots from it
>> >>>>>>>>>>> - "only" missing: get last db seq num
>> >>>>>>>>>>> - write new header
>> >>>>>>>>>>> - and deal with the local docs btree (if exists)
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks!
>> >>>>>>>>>>> Jan
>> >>>>>>>>>>> --
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> Filipe David Manana,
>> >>>>>>>>> fdmanana@apache.org
>> >>>>>>>>>
>> >>>>>>>>> "Reasonable men adapt themselves to the world.
>> >>>>>>>>> Unreasonable men adapt the world to themselves.
>> >>>>>>>>> That's why all progress depends on unreasonable men."
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Filipe David Manana,
>> >>>>>>> fdmanana@apache.org
>> >>>>>>>
>> >>>>>>> "Reasonable men adapt themselves to the world.
>> >>>>>>> Unreasonable men adapt the world to themselves.
>> >>>>>>> That's why all progress depends on unreasonable men."
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> >
>>
>>
>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
I think I found a bug in the current lost+found repair.
I've been running it against the testwritesdb and it's in a state that is
never finishing.
It's still spitting out these lines:
[info] [<0.32.0>] writing 1001 updates to lost+found/testwritesdb
Most are 1001 but there are also other random variances 452, 866, etc.
But the file size and dbinfo hasn't budged in over 30 minutes. The size is
stuck at 34300002 with the original db file being 54857478 .
This database only has one document in it that isn't "lost" so if it's
finding *any* new docs it should be writing them.
I also started another job to recover a production db that is quite large,
500megs, with the missing data a week or so back. This has been running for
2 hours and has still not output anything or created the lost and found db
so I can only assume that it is in the same state.
Both machines are still churning 100% CPU.
-Mikeal
On Mon, Aug 9, 2010 at 11:26 PM, Adam Kocoloski <ko...@apache.org> wrote:
> With Randall's help we hooked the new node scanner up to the lost+found DB
> generator. It seems to work well enough for small DBs; for large DBs with
> lots of missing nodes the O(N^2) complexity of the problem catches up to the
> code and generating the lost+found DB takes quite some time. Mikeal is
> running tests tonight. The algo appears pretty CPU-limited, so a little
> parallelization may be warranted.
>
> http://github.com/kocolosk/couchdb/tree/db_repair
>
> Adam
>
> (I sent this previous update to myself instead of the list, so I'll forward
> it here ...)
>
> On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
>
> > On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
> >
> >> Right, make_lost_and_found still relies on code which reads through
> couch_file one byte at a time, that's the cause of the slowness. The newer
> scanner will improve that pretty dramatically, and we can tune it further by
> increasing the length of the pattern that we match when looking for
> kp/kv_node terms in the files, at the expense of some extra complexity
> dealing with the block prefixes (currently it does a 1-byte match, which as
> I understand it cannot be split across blocks).
> >
> > The scanner now looks for a 7 byte match, unless it is within 6 bytes of
> a block boundary, in which case it looks for the longest possible match at
> that position. The more specific match condition greatly reduces the # of
> calls to couch_file, and thus boosts the throughput. On my laptop it can
> scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18
> seconds.
> >
> >> Regarding the file_corruption error on the larger file, I think this is
> something we will just naturally trigger when we take a guess that random
> positions in a file are actually the beginning of a term. I think our best
> recourse here is to return {error, file_corruption} from couch_file but
> leave the gen_server up and running instead of terminating it. That way the
> repair code can ignore the error and keep moving without having to reopen
> the file.
> >
> > I committed this change (to my db_repair branch) after consulting with
> Chris. The longer match condition makes these spurious file_corruption
> triggers much less likely, but I think it's still a good thing not to crash
> the server when they happen.
> >
> >> Next steps as I understand them - Randall is working on integrating the
> in-memory scanner into Volker's code that finds all the dangling by_id
> nodes. I'm working on making sure that the scanner identifies bt node
> candidates which span block prefixes, and on improving its pattern-matching.
> >
> > Latest from my end
> > http://github.com/kocolosk/couchdb/tree/db_repair
> >
> >>
> >> Adam
> >>
> >> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
> >>
> >>> I pulled down the latest code from Adam's branch @
> >>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
> >>>
> >>> Running timer:tc(couch_db_repair, make_lost_and_found,
> ["multi_conflict"]).
> >>> on a database with 200 lost updates spanning 200 restarts (
> >>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch )
> took
> >>> about 101 seconds.
> >>>
> >>> I tried running against a larger databases (
> >>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch )
> and I
> >>> got this exception:
> >>>
> >>> http://gist.github.com/516491
> >>>
> >>> -Mikeal
> >>>
> >>>
> >>>
> >>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <randall.leeds@gmail.com
> >wrote:
> >>>
> >>>> Summing up what went on in IRC for those who were absent.
> >>>>
> >>>> The latest progress is on Adam's branch at
> >>>> http://github.com/kocolosk/couchdb/tree/db_repair
> >>>>
> >>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
> >>>> lost+found/DbName database to which it merges all nodes not accessible
> >>>> from anywhere (any other node found in a full file scan or any header
> >>>> pointers).
> >>>>
> >>>> Currently, make_lost_and_found uses Volker's repair (from
> >>>> couch_db_repair_b module, also in Adam's branch).
> >>>> Adam found that the bottleneck was couch_file calls and that the
> >>>> repair process was taking a very long time so he added
> >>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
> >>>> and tries to process it to find nodes instead of scanning back one
> >>>> byte at a time. It is currently not hooked up to the repair mechanism.
> >>>>
> >>>> Making progress. Go team.
> >>>>
> >>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
> >>>> wrote:
> >>>>> jchris suggested on IRC that I try a normal doc update and see if
> that
> >>>> fixes
> >>>>> it.
> >>>>>
> >>>>> It does. After a new doc was created the dbinfo doc count was back to
> >>>>> normal.
> >>>>>
> >>>>> -Mikeal
> >>>>>
> >>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <
> mikeal.rogers@gmail.com
> >>>>> wrote:
> >>>>>
> >>>>>> Ok, I pulled down this code and tested against a database with a ton
> of
> >>>>>> missing writes right before a single restart.
> >>>>>>
> >>>>>> Before restart this was the database:
> >>>>>>
> >>>>>> {
> >>>>>> db_name: "testwritesdb"
> >>>>>> doc_count: 124969
> >>>>>> doc_del_count: 0
> >>>>>> update_seq: 124969
> >>>>>> purge_seq: 0
> >>>>>> compact_running: false
> >>>>>> disk_size: 54857478
> >>>>>> instance_start_time: "1281384140058211"
> >>>>>> disk_format_version: 5
> >>>>>> }
> >>>>>>
> >>>>>> After restart it was this:
> >>>>>>
> >>>>>> {
> >>>>>> db_name: "testwritesdb"
> >>>>>> doc_count: 1
> >>>>>> doc_del_count: 0
> >>>>>> update_seq: 1
> >>>>>> purge_seq: 0
> >>>>>> compact_running: false
> >>>>>> disk_size: 54857478
> >>>>>> instance_start_time: "1281384593876026"
> >>>>>> disk_format_version: 5
> >>>>>> }
> >>>>>>
> >>>>>> After repair, it's this:
> >>>>>>
> >>>>>> {
> >>>>>> db_name: "testwritesdb"
> >>>>>> doc_count: 1
> >>>>>> doc_del_count: 0
> >>>>>> update_seq: 124969
> >>>>>> purge_seq: 0
> >>>>>> compact_running: false
> >>>>>> disk_size: 54857820
> >>>>>> instance_start_time: "1281385990193289"
> >>>>>> disk_format_version: 5
> >>>>>> committed_update_seq: 124969
> >>>>>> }
> >>>>>>
> >>>>>> All the sequences are there and hitting _all_docs shows all the
> >>>> documents
> >>>>>> so why is the doc_count only 1 in the dbinfo?
> >>>>>>
> >>>>>> -Mikeal
> >>>>>>
> >>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
> >>>> fdmanana@apache.org>wrote:
> >>>>>>
> >>>>>>> For the record (and people not on IRC), the code at:
> >>>>>>>
> >>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
> >>>>>>>
> >>>>>>> is working for at least simple cases. Use
> >>>>>>> couch_db_repair:repair(DbNameAsString).
> >>>>>>> There's one TODO: update the reduce values for the by_seq and
> by_id
> >>>>>>> BTrees.
> >>>>>>>
> >>>>>>> If anyone wants to give some help on this, your welcome.
> >>>>>>>
> >>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <
> mikeal.rogers@gmail.com
> >>>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I'm starting to create a bunch of test db files that expose this
> bug
> >>>>>>> under
> >>>>>>>> different conditions like multiple restarts, across compaction,
> >>>>>>> variances
> >>>>>>>> in
> >>>>>>>> updates the might cause conflict, etc.
> >>>>>>>>
> >>>>>>>> http://github.com/mikeal/couchtest
> >>>>>>>>
> >>>>>>>> The README outlines what was done to the db's and what needs to be
> >>>>>>>> recovered.
> >>>>>>>>
> >>>>>>>> -Mikeal
> >>>>>>>>
> >>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
> >>>>>>> fdmanana@apache.org
> >>>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
> >>>>>>> robert.newson@gmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Doesn't this bit;
> >>>>>>>>>>
> >>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
> >>>>>>>>>> + Db;
> >>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
> >>>>>>>>>>
> >>>>>>>>>> revert the bug fix?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's intentional, for my local testing.
> >>>>>>>>> That patch isn't obviously anything close to final, it's too
> >>>>>>> experimental
> >>>>>>>>> yet.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> B.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>> Hi All,
> >>>>>>>>>>>
> >>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but he
> >>>>>>> isn't
> >>>>>>>>> done
> >>>>>>>>>> yet.
> >>>>>>>>>>>
> >>>>>>>>>>> Here's the current patch:
> >>>>>>>>>>>
> >>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> >>>>>>>>>>>
> >>>>>>>>>>> it is not done and very early, but any help on this is greatly
> >>>>>>>>>> appreciated.
> >>>>>>>>>>>
> >>>>>>>>>>> The current state is (in Filipe's words):
> >>>>>>>>>>> - i can detect that a file needs repair
> >>>>>>>>>>> - and get the last btree roots from it
> >>>>>>>>>>> - "only" missing: get last db seq num
> >>>>>>>>>>> - write new header
> >>>>>>>>>>> - and deal with the local docs btree (if exists)
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks!
> >>>>>>>>>>> Jan
> >>>>>>>>>>> --
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Filipe David Manana,
> >>>>>>>>> fdmanana@apache.org
> >>>>>>>>>
> >>>>>>>>> "Reasonable men adapt themselves to the world.
> >>>>>>>>> Unreasonable men adapt the world to themselves.
> >>>>>>>>> That's why all progress depends on unreasonable men."
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Filipe David Manana,
> >>>>>>> fdmanana@apache.org
> >>>>>>>
> >>>>>>> "Reasonable men adapt themselves to the world.
> >>>>>>> Unreasonable men adapt the world to themselves.
> >>>>>>> That's why all progress depends on unreasonable men."
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >
>
>
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
With Randall's help we hooked the new node scanner up to the lost+found DB generator. It seems to work well enough for small DBs; for large DBs with lots of missing nodes the O(N^2) complexity of the problem catches up to the code and generating the lost+found DB takes quite some time. Mikeal is running tests tonight. The algo appears pretty CPU-limited, so a little parallelization may be warranted.
http://github.com/kocolosk/couchdb/tree/db_repair
Adam
(I sent this previous update to myself instead of the list, so I'll forward it here ...)
On Aug 10, 2010, at 12:01 AM, Adam Kocoloski wrote:
> On Aug 9, 2010, at 10:10 PM, Adam Kocoloski wrote:
>
>> Right, make_lost_and_found still relies on code which reads through couch_file one byte at a time, that's the cause of the slowness. The newer scanner will improve that pretty dramatically, and we can tune it further by increasing the length of the pattern that we match when looking for kp/kv_node terms in the files, at the expense of some extra complexity dealing with the block prefixes (currently it does a 1-byte match, which as I understand it cannot be split across blocks).
>
> The scanner now looks for a 7 byte match, unless it is within 6 bytes of a block boundary, in which case it looks for the longest possible match at that position. The more specific match condition greatly reduces the # of calls to couch_file, and thus boosts the throughput. On my laptop it can scan the testwritesdb.couch from Mikeal's couchtest repo (52 MB) in 18 seconds.
>
>> Regarding the file_corruption error on the larger file, I think this is something we will just naturally trigger when we take a guess that random positions in a file are actually the beginning of a term. I think our best recourse here is to return {error, file_corruption} from couch_file but leave the gen_server up and running instead of terminating it. That way the repair code can ignore the error and keep moving without having to reopen the file.
>
> I committed this change (to my db_repair branch) after consulting with Chris. The longer match condition makes these spurious file_corruption triggers much less likely, but I think it's still a good thing not to crash the server when they happen.
>
>> Next steps as I understand them - Randall is working on integrating the in-memory scanner into Volker's code that finds all the dangling by_id nodes. I'm working on making sure that the scanner identifies bt node candidates which span block prefixes, and on improving its pattern-matching.
>
> Latest from my end
> http://github.com/kocolosk/couchdb/tree/db_repair
>
>>
>> Adam
>>
>> On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
>>
>>> I pulled down the latest code from Adam's branch @
>>> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>>>
>>> Running timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
>>> on a database with 200 lost updates spanning 200 restarts (
>>> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch ) took
>>> about 101 seconds.
>>>
>>> I tried running against a larger databases (
>>> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch ) and I
>>> got this exception:
>>>
>>> http://gist.github.com/516491
>>>
>>> -Mikeal
>>>
>>>
>>>
>>> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <ra...@gmail.com>wrote:
>>>
>>>> Summing up what went on in IRC for those who were absent.
>>>>
>>>> The latest progress is on Adam's branch at
>>>> http://github.com/kocolosk/couchdb/tree/db_repair
>>>>
>>>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>>>> lost+found/DbName database to which it merges all nodes not accessible
>>>> from anywhere (any other node found in a full file scan or any header
>>>> pointers).
>>>>
>>>> Currently, make_lost_and_found uses Volker's repair (from
>>>> couch_db_repair_b module, also in Adam's branch).
>>>> Adam found that the bottleneck was couch_file calls and that the
>>>> repair process was taking a very long time so he added
>>>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
>>>> and tries to process it to find nodes instead of scanning back one
>>>> byte at a time. It is currently not hooked up to the repair mechanism.
>>>>
>>>> Making progress. Go team.
>>>>
>>>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
>>>> wrote:
>>>>> jchris suggested on IRC that I try a normal doc update and see if that
>>>> fixes
>>>>> it.
>>>>>
>>>>> It does. After a new doc was created the dbinfo doc count was back to
>>>>> normal.
>>>>>
>>>>> -Mikeal
>>>>>
>>>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <mikeal.rogers@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Ok, I pulled down this code and tested against a database with a ton of
>>>>>> missing writes right before a single restart.
>>>>>>
>>>>>> Before restart this was the database:
>>>>>>
>>>>>> {
>>>>>> db_name: "testwritesdb"
>>>>>> doc_count: 124969
>>>>>> doc_del_count: 0
>>>>>> update_seq: 124969
>>>>>> purge_seq: 0
>>>>>> compact_running: false
>>>>>> disk_size: 54857478
>>>>>> instance_start_time: "1281384140058211"
>>>>>> disk_format_version: 5
>>>>>> }
>>>>>>
>>>>>> After restart it was this:
>>>>>>
>>>>>> {
>>>>>> db_name: "testwritesdb"
>>>>>> doc_count: 1
>>>>>> doc_del_count: 0
>>>>>> update_seq: 1
>>>>>> purge_seq: 0
>>>>>> compact_running: false
>>>>>> disk_size: 54857478
>>>>>> instance_start_time: "1281384593876026"
>>>>>> disk_format_version: 5
>>>>>> }
>>>>>>
>>>>>> After repair, it's this:
>>>>>>
>>>>>> {
>>>>>> db_name: "testwritesdb"
>>>>>> doc_count: 1
>>>>>> doc_del_count: 0
>>>>>> update_seq: 124969
>>>>>> purge_seq: 0
>>>>>> compact_running: false
>>>>>> disk_size: 54857820
>>>>>> instance_start_time: "1281385990193289"
>>>>>> disk_format_version: 5
>>>>>> committed_update_seq: 124969
>>>>>> }
>>>>>>
>>>>>> All the sequences are there and hitting _all_docs shows all the
>>>> documents
>>>>>> so why is the doc_count only 1 in the dbinfo?
>>>>>>
>>>>>> -Mikeal
>>>>>>
>>>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>>>> fdmanana@apache.org>wrote:
>>>>>>
>>>>>>> For the record (and people not on IRC), the code at:
>>>>>>>
>>>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>>>>>>>
>>>>>>> is working for at least simple cases. Use
>>>>>>> couch_db_repair:repair(DbNameAsString).
>>>>>>> There's one TODO: update the reduce values for the by_seq and by_id
>>>>>>> BTrees.
>>>>>>>
>>>>>>> If anyone wants to give some help on this, your welcome.
>>>>>>>
>>>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mikeal.rogers@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'm starting to create a bunch of test db files that expose this bug
>>>>>>> under
>>>>>>>> different conditions like multiple restarts, across compaction,
>>>>>>> variances
>>>>>>>> in
>>>>>>>> updates the might cause conflict, etc.
>>>>>>>>
>>>>>>>> http://github.com/mikeal/couchtest
>>>>>>>>
>>>>>>>> The README outlines what was done to the db's and what needs to be
>>>>>>>> recovered.
>>>>>>>>
>>>>>>>> -Mikeal
>>>>>>>>
>>>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>>>>>>> fdmanana@apache.org
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>>>>>>> robert.newson@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Doesn't this bit;
>>>>>>>>>>
>>>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>>>>>>>>>> + Db;
>>>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>>>>>>>>>>
>>>>>>>>>> revert the bug fix?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That's intentional, for my local testing.
>>>>>>>>> That patch isn't obviously anything close to final, it's too
>>>>>>> experimental
>>>>>>>>> yet.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> B.
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> Filipe jumped in to start working on the recovery tool, but he
>>>>>>> isn't
>>>>>>>>> done
>>>>>>>>>> yet.
>>>>>>>>>>>
>>>>>>>>>>> Here's the current patch:
>>>>>>>>>>>
>>>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>>>>>>>>>>
>>>>>>>>>>> it is not done and very early, but any help on this is greatly
>>>>>>>>>> appreciated.
>>>>>>>>>>>
>>>>>>>>>>> The current state is (in Filipe's words):
>>>>>>>>>>> - i can detect that a file needs repair
>>>>>>>>>>> - and get the last btree roots from it
>>>>>>>>>>> - "only" missing: get last db seq num
>>>>>>>>>>> - write new header
>>>>>>>>>>> - and deal with the local docs btree (if exists)
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> Jan
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Filipe David Manana,
>>>>>>>>> fdmanana@apache.org
>>>>>>>>>
>>>>>>>>> "Reasonable men adapt themselves to the world.
>>>>>>>>> Unreasonable men adapt the world to themselves.
>>>>>>>>> That's why all progress depends on unreasonable men."
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Filipe David Manana,
>>>>>>> fdmanana@apache.org
>>>>>>>
>>>>>>> "Reasonable men adapt themselves to the world.
>>>>>>> Unreasonable men adapt the world to themselves.
>>>>>>> That's why all progress depends on unreasonable men."
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: data recovery tool progress
Posted by Adam Kocoloski <ko...@apache.org>.
Right, make_lost_and_found still relies on code which reads through couch_file one byte at a time, that's the cause of the slowness. The newer scanner will improve that pretty dramatically, and we can tune it further by increasing the length of the pattern that we match when looking for kp/kv_node terms in the files, at the expense of some extra complexity dealing with the block prefixes (currently it does a 1-byte match, which as I understand it cannot be split across blocks).
Regarding the file_corruption error on the larger file, I think this is something we will just naturally trigger when we take a guess that random positions in a file are actually the beginning of a term. I think our best recourse here is to return {error, file_corruption} from couch_file but leave the gen_server up and running instead of terminating it. That way the repair code can ignore the error and keep moving without having to reopen the file.
Next steps as I understand them - Randall is working on integrating the in-memory scanner into Volker's code that finds all the dangling by_id nodes. I'm working on making sure that the scanner identifies bt node candidates which span block prefixes, and on improving its pattern-matching.
Adam
On Aug 9, 2010, at 9:50 PM, Mikeal Rogers wrote:
> I pulled down the latest code from Adam's branch @
> 7080ff72baa329cf6c4be2a79e71a41f744ed93b.
>
> Running timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
> on a database with 200 lost updates spanning 200 restarts (
> http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch ) took
> about 101 seconds.
>
> I tried running against a larger databases (
> http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch ) and I
> got this exception:
>
> http://gist.github.com/516491
>
> -Mikeal
>
>
>
> On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <ra...@gmail.com>wrote:
>
>> Summing up what went on in IRC for those who were absent.
>>
>> The latest progress is on Adam's branch at
>> http://github.com/kocolosk/couchdb/tree/db_repair
>>
>> couch_db_repair:make_lost_and_found/1 attempts to create a new
>> lost+found/DbName database to which it merges all nodes not accessible
>> from anywhere (any other node found in a full file scan or any header
>> pointers).
>>
>> Currently, make_lost_and_found uses Volker's repair (from
>> couch_db_repair_b module, also in Adam's branch).
>> Adam found that the bottleneck was couch_file calls and that the
>> repair process was taking a very long time so he added
>> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
>> and tries to process it to find nodes instead of scanning back one
>> byte at a time. It is currently not hooked up to the repair mechanism.
>>
>> Making progress. Go team.
>>
>> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
>> wrote:
>>> jchris suggested on IRC that I try a normal doc update and see if that
>> fixes
>>> it.
>>>
>>> It does. After a new doc was created the dbinfo doc count was back to
>>> normal.
>>>
>>> -Mikeal
>>>
>>> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <mikeal.rogers@gmail.com
>>> wrote:
>>>
>>>> Ok, I pulled down this code and tested against a database with a ton of
>>>> missing writes right before a single restart.
>>>>
>>>> Before restart this was the database:
>>>>
>>>> {
>>>> db_name: "testwritesdb"
>>>> doc_count: 124969
>>>> doc_del_count: 0
>>>> update_seq: 124969
>>>> purge_seq: 0
>>>> compact_running: false
>>>> disk_size: 54857478
>>>> instance_start_time: "1281384140058211"
>>>> disk_format_version: 5
>>>> }
>>>>
>>>> After restart it was this:
>>>>
>>>> {
>>>> db_name: "testwritesdb"
>>>> doc_count: 1
>>>> doc_del_count: 0
>>>> update_seq: 1
>>>> purge_seq: 0
>>>> compact_running: false
>>>> disk_size: 54857478
>>>> instance_start_time: "1281384593876026"
>>>> disk_format_version: 5
>>>> }
>>>>
>>>> After repair, it's this:
>>>>
>>>> {
>>>> db_name: "testwritesdb"
>>>> doc_count: 1
>>>> doc_del_count: 0
>>>> update_seq: 124969
>>>> purge_seq: 0
>>>> compact_running: false
>>>> disk_size: 54857820
>>>> instance_start_time: "1281385990193289"
>>>> disk_format_version: 5
>>>> committed_update_seq: 124969
>>>> }
>>>>
>>>> All the sequences are there and hitting _all_docs shows all the
>> documents
>>>> so why is the doc_count only 1 in the dbinfo?
>>>>
>>>> -Mikeal
>>>>
>>>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
>> fdmanana@apache.org>wrote:
>>>>
>>>>> For the record (and people not on IRC), the code at:
>>>>>
>>>>> http://github.com/fdmanana/couchdb/commits/db_repair
>>>>>
>>>>> is working for at least simple cases. Use
>>>>> couch_db_repair:repair(DbNameAsString).
>>>>> There's one TODO: update the reduce values for the by_seq and by_id
>>>>> BTrees.
>>>>>
>>>>> If anyone wants to give some help on this, your welcome.
>>>>>
>>>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mikeal.rogers@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> I'm starting to create a bunch of test db files that expose this bug
>>>>> under
>>>>>> different conditions like multiple restarts, across compaction,
>>>>> variances
>>>>>> in
>>>>>> updates the might cause conflict, etc.
>>>>>>
>>>>>> http://github.com/mikeal/couchtest
>>>>>>
>>>>>> The README outlines what was done to the db's and what needs to be
>>>>>> recovered.
>>>>>>
>>>>>> -Mikeal
>>>>>>
>>>>>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>>>>> fdmanana@apache.org
>>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>>>>> robert.newson@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> Doesn't this bit;
>>>>>>>>
>>>>>>>> - Db#db{waiting_delayed_commit=nil};
>>>>>>>> + Db;
>>>>>>>> + % Db#db{waiting_delayed_commit=nil};
>>>>>>>>
>>>>>>>> revert the bug fix?
>>>>>>>>
>>>>>>>
>>>>>>> That's intentional, for my local testing.
>>>>>>> That patch isn't obviously anything close to final, it's too
>>>>> experimental
>>>>>>> yet.
>>>>>>>
>>>>>>>>
>>>>>>>> B.
>>>>>>>>
>>>>>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>>>>> wrote:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> Filipe jumped in to start working on the recovery tool, but he
>>>>> isn't
>>>>>>> done
>>>>>>>> yet.
>>>>>>>>>
>>>>>>>>> Here's the current patch:
>>>>>>>>>
>>>>>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>>>>>>>>
>>>>>>>>> it is not done and very early, but any help on this is greatly
>>>>>>>> appreciated.
>>>>>>>>>
>>>>>>>>> The current state is (in Filipe's words):
>>>>>>>>> - i can detect that a file needs repair
>>>>>>>>> - and get the last btree roots from it
>>>>>>>>> - "only" missing: get last db seq num
>>>>>>>>> - write new header
>>>>>>>>> - and deal with the local docs btree (if exists)
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Jan
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Filipe David Manana,
>>>>>>> fdmanana@apache.org
>>>>>>>
>>>>>>> "Reasonable men adapt themselves to the world.
>>>>>>> Unreasonable men adapt the world to themselves.
>>>>>>> That's why all progress depends on unreasonable men."
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Filipe David Manana,
>>>>> fdmanana@apache.org
>>>>>
>>>>> "Reasonable men adapt themselves to the world.
>>>>> Unreasonable men adapt the world to themselves.
>>>>> That's why all progress depends on unreasonable men."
>>>>>
>>>>
>>>>
>>>
>>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
I pulled down the latest code from Adam's branch @
7080ff72baa329cf6c4be2a79e71a41f744ed93b.
Running timer:tc(couch_db_repair, make_lost_and_found, ["multi_conflict"]).
on a database with 200 lost updates spanning 200 restarts (
http://github.com/mikeal/couchtest/blob/master/multi_conflict.couch ) took
about 101 seconds.
I tried running against a larger databases (
http://github.com/mikeal/couchtest/blob/master/testwritesdb.couch ) and I
got this exception:
http://gist.github.com/516491
-Mikeal
On Mon, Aug 9, 2010 at 6:09 PM, Randall Leeds <ra...@gmail.com>wrote:
> Summing up what went on in IRC for those who were absent.
>
> The latest progress is on Adam's branch at
> http://github.com/kocolosk/couchdb/tree/db_repair
>
> couch_db_repair:make_lost_and_found/1 attempts to create a new
> lost+found/DbName database to which it merges all nodes not accessible
> from anywhere (any other node found in a full file scan or any header
> pointers).
>
> Currently, make_lost_and_found uses Volker's repair (from
> couch_db_repair_b module, also in Adam's branch).
> Adam found that the bottleneck was couch_file calls and that the
> repair process was taking a very long time so he added
> couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
> and tries to process it to find nodes instead of scanning back one
> byte at a time. It is currently not hooked up to the repair mechanism.
>
> Making progress. Go team.
>
> On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com>
> wrote:
> > jchris suggested on IRC that I try a normal doc update and see if that
> fixes
> > it.
> >
> > It does. After a new doc was created the dbinfo doc count was back to
> > normal.
> >
> > -Mikeal
> >
> > On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <mikeal.rogers@gmail.com
> >wrote:
> >
> >> Ok, I pulled down this code and tested against a database with a ton of
> >> missing writes right before a single restart.
> >>
> >> Before restart this was the database:
> >>
> >> {
> >> db_name: "testwritesdb"
> >> doc_count: 124969
> >> doc_del_count: 0
> >> update_seq: 124969
> >> purge_seq: 0
> >> compact_running: false
> >> disk_size: 54857478
> >> instance_start_time: "1281384140058211"
> >> disk_format_version: 5
> >> }
> >>
> >> After restart it was this:
> >>
> >> {
> >> db_name: "testwritesdb"
> >> doc_count: 1
> >> doc_del_count: 0
> >> update_seq: 1
> >> purge_seq: 0
> >> compact_running: false
> >> disk_size: 54857478
> >> instance_start_time: "1281384593876026"
> >> disk_format_version: 5
> >> }
> >>
> >> After repair, it's this:
> >>
> >> {
> >> db_name: "testwritesdb"
> >> doc_count: 1
> >> doc_del_count: 0
> >> update_seq: 124969
> >> purge_seq: 0
> >> compact_running: false
> >> disk_size: 54857820
> >> instance_start_time: "1281385990193289"
> >> disk_format_version: 5
> >> committed_update_seq: 124969
> >> }
> >>
> >> All the sequences are there and hitting _all_docs shows all the
> documents
> >> so why is the doc_count only 1 in the dbinfo?
> >>
> >> -Mikeal
> >>
> >> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <
> fdmanana@apache.org>wrote:
> >>
> >>> For the record (and people not on IRC), the code at:
> >>>
> >>> http://github.com/fdmanana/couchdb/commits/db_repair
> >>>
> >>> is working for at least simple cases. Use
> >>> couch_db_repair:repair(DbNameAsString).
> >>> There's one TODO: update the reduce values for the by_seq and by_id
> >>> BTrees.
> >>>
> >>> If anyone wants to give some help on this, your welcome.
> >>>
> >>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mikeal.rogers@gmail.com
> >>> >wrote:
> >>>
> >>> > I'm starting to create a bunch of test db files that expose this bug
> >>> under
> >>> > different conditions like multiple restarts, across compaction,
> >>> variances
> >>> > in
> >>> > updates the might cause conflict, etc.
> >>> >
> >>> > http://github.com/mikeal/couchtest
> >>> >
> >>> > The README outlines what was done to the db's and what needs to be
> >>> > recovered.
> >>> >
> >>> > -Mikeal
> >>> >
> >>> > On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
> >>> fdmanana@apache.org
> >>> > >wrote:
> >>> >
> >>> > > On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
> >>> robert.newson@gmail.com
> >>> > > >wrote:
> >>> > >
> >>> > > > Doesn't this bit;
> >>> > > >
> >>> > > > - Db#db{waiting_delayed_commit=nil};
> >>> > > > + Db;
> >>> > > > + % Db#db{waiting_delayed_commit=nil};
> >>> > > >
> >>> > > > revert the bug fix?
> >>> > > >
> >>> > >
> >>> > > That's intentional, for my local testing.
> >>> > > That patch isn't obviously anything close to final, it's too
> >>> experimental
> >>> > > yet.
> >>> > >
> >>> > > >
> >>> > > > B.
> >>> > > >
> >>> > > > On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
> >>> wrote:
> >>> > > > > Hi All,
> >>> > > > >
> >>> > > > > Filipe jumped in to start working on the recovery tool, but he
> >>> isn't
> >>> > > done
> >>> > > > yet.
> >>> > > > >
> >>> > > > > Here's the current patch:
> >>> > > > >
> >>> > > > > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> >>> > > > >
> >>> > > > > it is not done and very early, but any help on this is greatly
> >>> > > > appreciated.
> >>> > > > >
> >>> > > > > The current state is (in Filipe's words):
> >>> > > > > - i can detect that a file needs repair
> >>> > > > > - and get the last btree roots from it
> >>> > > > > - "only" missing: get last db seq num
> >>> > > > > - write new header
> >>> > > > > - and deal with the local docs btree (if exists)
> >>> > > > >
> >>> > > > > Thanks!
> >>> > > > > Jan
> >>> > > > > --
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Filipe David Manana,
> >>> > > fdmanana@apache.org
> >>> > >
> >>> > > "Reasonable men adapt themselves to the world.
> >>> > > Unreasonable men adapt the world to themselves.
> >>> > > That's why all progress depends on unreasonable men."
> >>> > >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Filipe David Manana,
> >>> fdmanana@apache.org
> >>>
> >>> "Reasonable men adapt themselves to the world.
> >>> Unreasonable men adapt the world to themselves.
> >>> That's why all progress depends on unreasonable men."
> >>>
> >>
> >>
> >
>
Re: data recovery tool progress
Posted by Randall Leeds <ra...@gmail.com>.
Summing up what went on in IRC for those who were absent.
The latest progress is on Adam's branch at
http://github.com/kocolosk/couchdb/tree/db_repair
couch_db_repair:make_lost_and_found/1 attempts to create a new
lost+found/DbName database to which it merges all nodes not accessible
from anywhere (any other node found in a full file scan or any header
pointers).
Currently, make_lost_and_found uses Volker's repair (from
couch_db_repair_b module, also in Adam's branch).
Adam found that the bottleneck was couch_file calls and that the
repair process was taking a very long time so he added
couch_db_repair:find_nodes_quickly/1 that reads 1MB chunks as binary
and tries to process it to find nodes instead of scanning back one
byte at a time. It is currently not hooked up to the repair mechanism.
Making progress. Go team.
On Mon, Aug 9, 2010 at 13:52, Mikeal Rogers <mi...@gmail.com> wrote:
> jchris suggested on IRC that I try a normal doc update and see if that fixes
> it.
>
> It does. After a new doc was created the dbinfo doc count was back to
> normal.
>
> -Mikeal
>
> On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <mi...@gmail.com>wrote:
>
>> Ok, I pulled down this code and tested against a database with a ton of
>> missing writes right before a single restart.
>>
>> Before restart this was the database:
>>
>> {
>> db_name: "testwritesdb"
>> doc_count: 124969
>> doc_del_count: 0
>> update_seq: 124969
>> purge_seq: 0
>> compact_running: false
>> disk_size: 54857478
>> instance_start_time: "1281384140058211"
>> disk_format_version: 5
>> }
>>
>> After restart it was this:
>>
>> {
>> db_name: "testwritesdb"
>> doc_count: 1
>> doc_del_count: 0
>> update_seq: 1
>> purge_seq: 0
>> compact_running: false
>> disk_size: 54857478
>> instance_start_time: "1281384593876026"
>> disk_format_version: 5
>> }
>>
>> After repair, it's this:
>>
>> {
>> db_name: "testwritesdb"
>> doc_count: 1
>> doc_del_count: 0
>> update_seq: 124969
>> purge_seq: 0
>> compact_running: false
>> disk_size: 54857820
>> instance_start_time: "1281385990193289"
>> disk_format_version: 5
>> committed_update_seq: 124969
>> }
>>
>> All the sequences are there and hitting _all_docs shows all the documents
>> so why is the doc_count only 1 in the dbinfo?
>>
>> -Mikeal
>>
>> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <fd...@apache.org>wrote:
>>
>>> For the record (and people not on IRC), the code at:
>>>
>>> http://github.com/fdmanana/couchdb/commits/db_repair
>>>
>>> is working for at least simple cases. Use
>>> couch_db_repair:repair(DbNameAsString).
>>> There's one TODO: update the reduce values for the by_seq and by_id
>>> BTrees.
>>>
>>> If anyone wants to give some help on this, your welcome.
>>>
>>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mikeal.rogers@gmail.com
>>> >wrote:
>>>
>>> > I'm starting to create a bunch of test db files that expose this bug
>>> under
>>> > different conditions like multiple restarts, across compaction,
>>> variances
>>> > in
>>> > updates the might cause conflict, etc.
>>> >
>>> > http://github.com/mikeal/couchtest
>>> >
>>> > The README outlines what was done to the db's and what needs to be
>>> > recovered.
>>> >
>>> > -Mikeal
>>> >
>>> > On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>>> fdmanana@apache.org
>>> > >wrote:
>>> >
>>> > > On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>>> robert.newson@gmail.com
>>> > > >wrote:
>>> > >
>>> > > > Doesn't this bit;
>>> > > >
>>> > > > - Db#db{waiting_delayed_commit=nil};
>>> > > > + Db;
>>> > > > + % Db#db{waiting_delayed_commit=nil};
>>> > > >
>>> > > > revert the bug fix?
>>> > > >
>>> > >
>>> > > That's intentional, for my local testing.
>>> > > That patch isn't obviously anything close to final, it's too
>>> experimental
>>> > > yet.
>>> > >
>>> > > >
>>> > > > B.
>>> > > >
>>> > > > On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>>> wrote:
>>> > > > > Hi All,
>>> > > > >
>>> > > > > Filipe jumped in to start working on the recovery tool, but he
>>> isn't
>>> > > done
>>> > > > yet.
>>> > > > >
>>> > > > > Here's the current patch:
>>> > > > >
>>> > > > > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>> > > > >
>>> > > > > it is not done and very early, but any help on this is greatly
>>> > > > appreciated.
>>> > > > >
>>> > > > > The current state is (in Filipe's words):
>>> > > > > - i can detect that a file needs repair
>>> > > > > - and get the last btree roots from it
>>> > > > > - "only" missing: get last db seq num
>>> > > > > - write new header
>>> > > > > - and deal with the local docs btree (if exists)
>>> > > > >
>>> > > > > Thanks!
>>> > > > > Jan
>>> > > > > --
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Filipe David Manana,
>>> > > fdmanana@apache.org
>>> > >
>>> > > "Reasonable men adapt themselves to the world.
>>> > > Unreasonable men adapt the world to themselves.
>>> > > That's why all progress depends on unreasonable men."
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> Filipe David Manana,
>>> fdmanana@apache.org
>>>
>>> "Reasonable men adapt themselves to the world.
>>> Unreasonable men adapt the world to themselves.
>>> That's why all progress depends on unreasonable men."
>>>
>>
>>
>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
jchris suggested on IRC that I try a normal doc update and see if that fixes
it.
It does. After a new doc was created the dbinfo doc count was back to
normal.
-Mikeal
On Mon, Aug 9, 2010 at 1:39 PM, Mikeal Rogers <mi...@gmail.com>wrote:
> Ok, I pulled down this code and tested against a database with a ton of
> missing writes right before a single restart.
>
> Before restart this was the database:
>
> {
> db_name: "testwritesdb"
> doc_count: 124969
> doc_del_count: 0
> update_seq: 124969
> purge_seq: 0
> compact_running: false
> disk_size: 54857478
> instance_start_time: "1281384140058211"
> disk_format_version: 5
> }
>
> After restart it was this:
>
> {
> db_name: "testwritesdb"
> doc_count: 1
> doc_del_count: 0
> update_seq: 1
> purge_seq: 0
> compact_running: false
> disk_size: 54857478
> instance_start_time: "1281384593876026"
> disk_format_version: 5
> }
>
> After repair, it's this:
>
> {
> db_name: "testwritesdb"
> doc_count: 1
> doc_del_count: 0
> update_seq: 124969
> purge_seq: 0
> compact_running: false
> disk_size: 54857820
> instance_start_time: "1281385990193289"
> disk_format_version: 5
> committed_update_seq: 124969
> }
>
> All the sequences are there and hitting _all_docs shows all the documents
> so why is the doc_count only 1 in the dbinfo?
>
> -Mikeal
>
> On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <fd...@apache.org>wrote:
>
>> For the record (and people not on IRC), the code at:
>>
>> http://github.com/fdmanana/couchdb/commits/db_repair
>>
>> is working for at least simple cases. Use
>> couch_db_repair:repair(DbNameAsString).
>> There's one TODO: update the reduce values for the by_seq and by_id
>> BTrees.
>>
>> If anyone wants to give some help on this, your welcome.
>>
>> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mikeal.rogers@gmail.com
>> >wrote:
>>
>> > I'm starting to create a bunch of test db files that expose this bug
>> under
>> > different conditions like multiple restarts, across compaction,
>> variances
>> > in
>> > updates the might cause conflict, etc.
>> >
>> > http://github.com/mikeal/couchtest
>> >
>> > The README outlines what was done to the db's and what needs to be
>> > recovered.
>> >
>> > -Mikeal
>> >
>> > On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <
>> fdmanana@apache.org
>> > >wrote:
>> >
>> > > On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <
>> robert.newson@gmail.com
>> > > >wrote:
>> > >
>> > > > Doesn't this bit;
>> > > >
>> > > > - Db#db{waiting_delayed_commit=nil};
>> > > > + Db;
>> > > > + % Db#db{waiting_delayed_commit=nil};
>> > > >
>> > > > revert the bug fix?
>> > > >
>> > >
>> > > That's intentional, for my local testing.
>> > > That patch isn't obviously anything close to final, it's too
>> experimental
>> > > yet.
>> > >
>> > > >
>> > > > B.
>> > > >
>> > > > On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org>
>> wrote:
>> > > > > Hi All,
>> > > > >
>> > > > > Filipe jumped in to start working on the recovery tool, but he
>> isn't
>> > > done
>> > > > yet.
>> > > > >
>> > > > > Here's the current patch:
>> > > > >
>> > > > > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>> > > > >
>> > > > > it is not done and very early, but any help on this is greatly
>> > > > appreciated.
>> > > > >
>> > > > > The current state is (in Filipe's words):
>> > > > > - i can detect that a file needs repair
>> > > > > - and get the last btree roots from it
>> > > > > - "only" missing: get last db seq num
>> > > > > - write new header
>> > > > > - and deal with the local docs btree (if exists)
>> > > > >
>> > > > > Thanks!
>> > > > > Jan
>> > > > > --
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Filipe David Manana,
>> > > fdmanana@apache.org
>> > >
>> > > "Reasonable men adapt themselves to the world.
>> > > Unreasonable men adapt the world to themselves.
>> > > That's why all progress depends on unreasonable men."
>> > >
>> >
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana@apache.org
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>
>
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
Ok, I pulled down this code and tested against a database with a ton of
missing writes right before a single restart.
Before restart this was the database:
{
db_name: "testwritesdb"
doc_count: 124969
doc_del_count: 0
update_seq: 124969
purge_seq: 0
compact_running: false
disk_size: 54857478
instance_start_time: "1281384140058211"
disk_format_version: 5
}
After restart it was this:
{
db_name: "testwritesdb"
doc_count: 1
doc_del_count: 0
update_seq: 1
purge_seq: 0
compact_running: false
disk_size: 54857478
instance_start_time: "1281384593876026"
disk_format_version: 5
}
After repair, it's this:
{
db_name: "testwritesdb"
doc_count: 1
doc_del_count: 0
update_seq: 124969
purge_seq: 0
compact_running: false
disk_size: 54857820
instance_start_time: "1281385990193289"
disk_format_version: 5
committed_update_seq: 124969
}
All the sequences are there and hitting _all_docs shows all the documents so
why is the doc_count only 1 in the dbinfo?
-Mikeal
On Mon, Aug 9, 2010 at 11:53 AM, Filipe David Manana <fd...@apache.org>wrote:
> For the record (and people not on IRC), the code at:
>
> http://github.com/fdmanana/couchdb/commits/db_repair
>
> is working for at least simple cases. Use
> couch_db_repair:repair(DbNameAsString).
> There's one TODO: update the reduce values for the by_seq and by_id
> BTrees.
>
> If anyone wants to give some help on this, your welcome.
>
> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mikeal.rogers@gmail.com
> >wrote:
>
> > I'm starting to create a bunch of test db files that expose this bug
> under
> > different conditions like multiple restarts, across compaction, variances
> > in
> > updates the might cause conflict, etc.
> >
> > http://github.com/mikeal/couchtest
> >
> > The README outlines what was done to the db's and what needs to be
> > recovered.
> >
> > -Mikeal
> >
> > On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <fdmanana@apache.org
> > >wrote:
> >
> > > On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <robert.newson@gmail.com
> > > >wrote:
> > >
> > > > Doesn't this bit;
> > > >
> > > > - Db#db{waiting_delayed_commit=nil};
> > > > + Db;
> > > > + % Db#db{waiting_delayed_commit=nil};
> > > >
> > > > revert the bug fix?
> > > >
> > >
> > > That's intentional, for my local testing.
> > > That patch isn't obviously anything close to final, it's too
> experimental
> > > yet.
> > >
> > > >
> > > > B.
> > > >
> > > > On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> > > > > Hi All,
> > > > >
> > > > > Filipe jumped in to start working on the recovery tool, but he
> isn't
> > > done
> > > > yet.
> > > > >
> > > > > Here's the current patch:
> > > > >
> > > > > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> > > > >
> > > > > it is not done and very early, but any help on this is greatly
> > > > appreciated.
> > > > >
> > > > > The current state is (in Filipe's words):
> > > > > - i can detect that a file needs repair
> > > > > - and get the last btree roots from it
> > > > > - "only" missing: get last db seq num
> > > > > - write new header
> > > > > - and deal with the local docs btree (if exists)
> > > > >
> > > > > Thanks!
> > > > > Jan
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Filipe David Manana,
> > > fdmanana@apache.org
> > >
> > > "Reasonable men adapt themselves to the world.
> > > Unreasonable men adapt the world to themselves.
> > > That's why all progress depends on unreasonable men."
> > >
> >
>
>
>
> --
> Filipe David Manana,
> fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
Re: data recovery tool progress
Posted by J Chris Anderson <jc...@apache.org>.
I have described a generalized repair algorithm which should work even for CouchDB's that have been restarted multiple times leaving orphaned data in the middle of the file (and valid headers after it)
Adam Kocoloski has offered to write the "for Node in BtreeNodes" part and Volker Mische and I are working on collecting the optimal set of BtreeNodes.
http://wiki.couchone.com/page/recovery-tool (see the last section "Generalized algorithm")
If you want to write any of this, also write it along side us and we will compare notes and help each other out on IRC.
Chris
On Aug 9, 2010, at 11:53 AM, Filipe David Manana wrote:
> For the record (and people not on IRC), the code at:
>
> http://github.com/fdmanana/couchdb/commits/db_repair
>
> is working for at least simple cases. Use
> couch_db_repair:repair(DbNameAsString).
> There's one TODO: update the reduce values for the by_seq and by_id BTrees.
>
> If anyone wants to give some help on this, your welcome.
>
> On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mi...@gmail.com>wrote:
>
>> I'm starting to create a bunch of test db files that expose this bug under
>> different conditions like multiple restarts, across compaction, variances
>> in
>> updates the might cause conflict, etc.
>>
>> http://github.com/mikeal/couchtest
>>
>> The README outlines what was done to the db's and what needs to be
>> recovered.
>>
>> -Mikeal
>>
>> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <fdmanana@apache.org
>>> wrote:
>>
>>> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <robert.newson@gmail.com
>>>> wrote:
>>>
>>>> Doesn't this bit;
>>>>
>>>> - Db#db{waiting_delayed_commit=nil};
>>>> + Db;
>>>> + % Db#db{waiting_delayed_commit=nil};
>>>>
>>>> revert the bug fix?
>>>>
>>>
>>> That's intentional, for my local testing.
>>> That patch isn't obviously anything close to final, it's too experimental
>>> yet.
>>>
>>>>
>>>> B.
>>>>
>>>> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
>>>>> Hi All,
>>>>>
>>>>> Filipe jumped in to start working on the recovery tool, but he isn't
>>> done
>>>> yet.
>>>>>
>>>>> Here's the current patch:
>>>>>
>>>>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>>>>
>>>>> it is not done and very early, but any help on this is greatly
>>>> appreciated.
>>>>>
>>>>> The current state is (in Filipe's words):
>>>>> - i can detect that a file needs repair
>>>>> - and get the last btree roots from it
>>>>> - "only" missing: get last db seq num
>>>>> - write new header
>>>>> - and deal with the local docs btree (if exists)
>>>>>
>>>>> Thanks!
>>>>> Jan
>>>>> --
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Filipe David Manana,
>>> fdmanana@apache.org
>>>
>>> "Reasonable men adapt themselves to the world.
>>> Unreasonable men adapt the world to themselves.
>>> That's why all progress depends on unreasonable men."
>>>
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
Re: data recovery tool progress
Posted by Filipe David Manana <fd...@apache.org>.
For the record (and people not on IRC), the code at:
http://github.com/fdmanana/couchdb/commits/db_repair
is working for at least simple cases. Use
couch_db_repair:repair(DbNameAsString).
There's one TODO: update the reduce values for the by_seq and by_id BTrees.
If anyone wants to give some help on this, your welcome.
On Mon, Aug 9, 2010 at 6:12 PM, Mikeal Rogers <mi...@gmail.com>wrote:
> I'm starting to create a bunch of test db files that expose this bug under
> different conditions like multiple restarts, across compaction, variances
> in
> updates the might cause conflict, etc.
>
> http://github.com/mikeal/couchtest
>
> The README outlines what was done to the db's and what needs to be
> recovered.
>
> -Mikeal
>
> On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <fdmanana@apache.org
> >wrote:
>
> > On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <robert.newson@gmail.com
> > >wrote:
> >
> > > Doesn't this bit;
> > >
> > > - Db#db{waiting_delayed_commit=nil};
> > > + Db;
> > > + % Db#db{waiting_delayed_commit=nil};
> > >
> > > revert the bug fix?
> > >
> >
> > That's intentional, for my local testing.
> > That patch isn't obviously anything close to final, it's too experimental
> > yet.
> >
> > >
> > > B.
> > >
> > > On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> > > > Hi All,
> > > >
> > > > Filipe jumped in to start working on the recovery tool, but he isn't
> > done
> > > yet.
> > > >
> > > > Here's the current patch:
> > > >
> > > > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> > > >
> > > > it is not done and very early, but any help on this is greatly
> > > appreciated.
> > > >
> > > > The current state is (in Filipe's words):
> > > > - i can detect that a file needs repair
> > > > - and get the last btree roots from it
> > > > - "only" missing: get last db seq num
> > > > - write new header
> > > > - and deal with the local docs btree (if exists)
> > > >
> > > > Thanks!
> > > > Jan
> > > > --
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Filipe David Manana,
> > fdmanana@apache.org
> >
> > "Reasonable men adapt themselves to the world.
> > Unreasonable men adapt the world to themselves.
> > That's why all progress depends on unreasonable men."
> >
>
--
Filipe David Manana,
fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: data recovery tool progress
Posted by Mikeal Rogers <mi...@gmail.com>.
I'm starting to create a bunch of test db files that expose this bug under
different conditions like multiple restarts, across compaction, variances in
updates the might cause conflict, etc.
http://github.com/mikeal/couchtest
The README outlines what was done to the db's and what needs to be
recovered.
-Mikeal
On Mon, Aug 9, 2010 at 9:33 AM, Filipe David Manana <fd...@apache.org>wrote:
> On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <robert.newson@gmail.com
> >wrote:
>
> > Doesn't this bit;
> >
> > - Db#db{waiting_delayed_commit=nil};
> > + Db;
> > + % Db#db{waiting_delayed_commit=nil};
> >
> > revert the bug fix?
> >
>
> That's intentional, for my local testing.
> That patch isn't obviously anything close to final, it's too experimental
> yet.
>
> >
> > B.
> >
> > On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> > > Hi All,
> > >
> > > Filipe jumped in to start working on the recovery tool, but he isn't
> done
> > yet.
> > >
> > > Here's the current patch:
> > >
> > > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> > >
> > > it is not done and very early, but any help on this is greatly
> > appreciated.
> > >
> > > The current state is (in Filipe's words):
> > > - i can detect that a file needs repair
> > > - and get the last btree roots from it
> > > - "only" missing: get last db seq num
> > > - write new header
> > > - and deal with the local docs btree (if exists)
> > >
> > > Thanks!
> > > Jan
> > > --
> > >
> > >
> >
>
>
>
> --
> Filipe David Manana,
> fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
Re: data recovery tool progress
Posted by Filipe David Manana <fd...@apache.org>.
On Mon, Aug 9, 2010 at 5:22 PM, Robert Newson <ro...@gmail.com>wrote:
> Doesn't this bit;
>
> - Db#db{waiting_delayed_commit=nil};
> + Db;
> + % Db#db{waiting_delayed_commit=nil};
>
> revert the bug fix?
>
That's intentional, for my local testing.
That patch isn't obviously anything close to final, it's too experimental
yet.
>
> B.
>
> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> > Hi All,
> >
> > Filipe jumped in to start working on the recovery tool, but he isn't done
> yet.
> >
> > Here's the current patch:
> >
> > http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
> >
> > it is not done and very early, but any help on this is greatly
> appreciated.
> >
> > The current state is (in Filipe's words):
> > - i can detect that a file needs repair
> > - and get the last btree roots from it
> > - "only" missing: get last db seq num
> > - write new header
> > - and deal with the local docs btree (if exists)
> >
> > Thanks!
> > Jan
> > --
> >
> >
>
--
Filipe David Manana,
fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: data recovery tool progress
Posted by Jan Lehnardt <ja...@apache.org>.
Chris put this up on github for all of us to look at.
http://github.com/jchris/couchdb/tree/repcommits
On 9 Aug 2010, at 18:22, Robert Newson wrote:
> Doesn't this bit;
>
> - Db#db{waiting_delayed_commit=nil};
> + Db;
> + % Db#db{waiting_delayed_commit=nil};
>
> revert the bug fix?
>
> B.
>
> On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
>> Hi All,
>>
>> Filipe jumped in to start working on the recovery tool, but he isn't done yet.
>>
>> Here's the current patch:
>>
>> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>>
>> it is not done and very early, but any help on this is greatly appreciated.
>>
>> The current state is (in Filipe's words):
>> - i can detect that a file needs repair
>> - and get the last btree roots from it
>> - "only" missing: get last db seq num
>> - write new header
>> - and deal with the local docs btree (if exists)
>>
>> Thanks!
>> Jan
>> --
>>
>>
Re: data recovery tool progress
Posted by Robert Newson <ro...@gmail.com>.
Doesn't this bit;
- Db#db{waiting_delayed_commit=nil};
+ Db;
+ % Db#db{waiting_delayed_commit=nil};
revert the bug fix?
B.
On Mon, Aug 9, 2010 at 5:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> Hi All,
>
> Filipe jumped in to start working on the recovery tool, but he isn't done yet.
>
> Here's the current patch:
>
> http://www.friendpaste.com/4uMngrym4r7Zz4R0ThSHbz
>
> it is not done and very early, but any help on this is greatly appreciated.
>
> The current state is (in Filipe's words):
> - i can detect that a file needs repair
> - and get the last btree roots from it
> - "only" missing: get last db seq num
> - write new header
> - and deal with the local docs btree (if exists)
>
> Thanks!
> Jan
> --
>
>