You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Sharath <sh...@gmail.com> on 2020/06/09 00:44:26 UTC
couchdb file corruption
Hi,
I'm encountering a file corruption in CouchDb 2.3.1 running on Ubuntu
18.04.1 LTS.
The disk store is an ext4 SSD.
I'm unable to access the view and couch logs shows the error below.
Is there a way of knowing which view is corrupt (I have a few databases).
I'm thinking of deleting all the views and recreating them - would that
work?
thanks!
error 1:
[emergency] 2020-06-09T00:33:10.122474Z couchdb@127.0.0.1 <0.1427.0>
-------- File corruption in <0.951.0> at position 2668802729
[error] 2020-06-09T00:33:10.123271Z couchdb@127.0.0.1 <0.1392.0> 9f5e486e35
rexi_server: from: couchdb@127.0.0.1(<0.758.0>) mfa: fabric_rpc:map_view/5
throw:{file_corruption,<<"file corruption">>}
[{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2020-06-09T00:33:10.123712Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
req_err(4089041121) file_corruption : file corruption
[<<"couch_mrview_util:get_view_index_state/5
L137">>,<<"couch_mrview_util:get_view/4 L81">>,<<"couch_mrview:query_view/6
L247">>,<<"rexi_server:init_p/3 L140">>]
[notice] 2020-06-09T00:33:10.124065Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
192.168.0.13:5984 192.168.0.8 admin GET
/stock/_design/company/_view/getallamexcompanies?reduce=false&skip=0&limit=101
500 ok 1111
[error] 2020-06-09T00:33:11.129465Z couchdb@127.0.0.1 <0.1419.0> --------
gen_server <0.1419.0> terminated with reason: no match of right hand value
eof at couch_file:read_raw_iolist_int/3(line:627) <=
couch_file:handle_call/3(line:449) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
<= proc_lib:init_p_do_apply/3(line:247)
last msg: {pread_iolist,1190244510}
state:
[{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7921>,106}},false,1767501252,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/80000000-9fffffff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"}]}]
extra: []
[error] 2020-06-09T00:33:11.132250Z couchdb@127.0.0.1 <0.1419.0> --------
CRASH REPORT Process (<0.1419.0>) with 1 neighbors exited with reason: no
match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
<= couch_file:handle_call/3(line:449) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
<= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
<= proc_lib:init_p_do_apply/3(line:247); initial_call:
{couch_file,init,['Argument__1']}, ancestors: [<0.1408.0>,<0.1407.0>],
messages: [], links: [<0.1408.0>], dictionary:
[{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7921>,106}},"/dat..."}},...],
trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
reductions: 573398
[error] 2020-06-09T00:33:11.132607Z couchdb@127.0.0.1 <0.1432.0> --------
gen_server <0.1432.0> terminated with reason: no match of right hand value
eof at couch_file:read_raw_iolist_int/3(line:627) <=
couch_file:handle_call/3(line:449) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
<= proc_lib:init_p_do_apply/3(line:247)
last msg:
{'EXIT',<0.1408.0>,{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
state: {st,<0.1408.0>,couch_mrview_index,undefined}
extra: []
error 2:
[error] 2020-06-09T00:40:25.580012Z couchdb@127.0.0.1 <0.3409.0> --------
gen_server <0.3409.0> terminated with reason: no match of right hand value
eof at couch_file:read_raw_iolist_int/3(line:627) <=
couch_file:handle_call/3(line:449) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
<= proc_lib:init_p_do_apply/3(line:247)
last msg: {pread_iolist,1223893408}
state:
[{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7858>,108}},false,1686143327,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/00000000-1fffffff/stockanalysis.1584663342_design/mrview/98efc890b71fe5bf74611ad0868f39bf.view"}]}]
extra: []
[error] 2020-06-09T00:40:25.580132Z couchdb@127.0.0.1 <0.260.0> --------
Compaction daemon - an error ocurred while compacting the view group
`company` from database
`shards/00000000-1fffffff/stockanalysis.1584663342`:
{{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
[error] 2020-06-09T00:40:25.580705Z couchdb@127.0.0.1 <0.3409.0> --------
CRASH REPORT Process (<0.3409.0>) with 1 neighbors exited with reason: no
match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
<= couch_file:handle_call/3(line:449) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
<= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
<= proc_lib:init_p_do_apply/3(line:247); initial_call:
{couch_file,init,['Argument__1']}, ancestors: [<0.3408.0>,<0.3407.0>],
messages: [], links: [<0.3408.0>], dictionary:
[{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7858>,108}},"/dat..."}},...],
trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
reductions: 627634
[error] 2020-06-09T00:40:25.581361Z couchdb@127.0.0.1 emulator --------
Error in process <0.3415.0> on node 'couchdb@127.0.0.1' with exit value:
{{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
Re: couchdb file corruption
Posted by Sharath <sh...@gmail.com>.
Thanks Kiril and Jan.
@Kiril - yes thats the stock db - has ~12 million documents
@Jan - I've restored from backup (learnt the lesson some time ago and the
database is replicated to two other machines).
The design document contains multiple views and generating them takes a
long time.
thanks for your help.
Sharath
On Tue, Jun 9, 2020 at 5:32 PM Jan Lehnardt <ja...@apache.org> wrote:
> Hi Sharath,
>
> the error you are seeing originates in CouchDB’s file checksumming
> feature[1]. When writing a block of data to disk, in either a
> database or a view index, CouchDB also stores a checksum of the
> block. When reading the block later, it checks if the stored
> checksum and the newly read block still match. This is to prevent
> hard drive errors sneaking into your database.
>
> In your case, a block in a view index doesn’t match its checksum
> anymore. The reasons for this could be manifold, but all of them
> will have resulted in a bit or more on your storage device having
> changed values behind CouchDB’s back.
>
> To get back to working views, you can delete the index files and
> rebuild them.
>
> But more importantly:
>
> 1. do triple check what could have caused that block to go bad.
> You don’t want to have this happen to your database files.
>
> 2. triple check your backups are up to date and restorable.
>
> [1]:
> https://github.com/apache/couchdb/blob/3505281559513e2922484ebf0996a8846dcc0a34/src/couch/src/couch_file.erl#L173-L188
>
> Best
> Jan
> —
>
> > On 9. Jun 2020, at 02:44, Sharath <sh...@gmail.com> wrote:
> >
> > Hi,
> >
> > I'm encountering a file corruption in CouchDb 2.3.1 running on Ubuntu
> > 18.04.1 LTS.
> >
> > The disk store is an ext4 SSD.
> >
> > I'm unable to access the view and couch logs shows the error below.
> >
> > Is there a way of knowing which view is corrupt (I have a few databases).
> >
> > I'm thinking of deleting all the views and recreating them - would that
> > work?
> >
> > thanks!
> >
> > error 1:
> > [emergency] 2020-06-09T00:33:10.122474Z couchdb@127.0.0.1 <0.1427.0>
> > -------- File corruption in <0.951.0> at position 2668802729
> > [error] 2020-06-09T00:33:10.123271Z couchdb@127.0.0.1 <0.1392.0>
> 9f5e486e35
> > rexi_server: from: couchdb@127.0.0.1(<0.758.0>) mfa:
> fabric_rpc:map_view/5
> > throw:{file_corruption,<<"file corruption">>}
> >
> [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
> > [error] 2020-06-09T00:33:10.123712Z couchdb@127.0.0.1 <0.758.0>
> 9f5e486e35
> > req_err(4089041121) file_corruption : file corruption
> > [<<"couch_mrview_util:get_view_index_state/5
> > L137">>,<<"couch_mrview_util:get_view/4
> L81">>,<<"couch_mrview:query_view/6
> > L247">>,<<"rexi_server:init_p/3 L140">>]
> > [notice] 2020-06-09T00:33:10.124065Z couchdb@127.0.0.1 <0.758.0>
> 9f5e486e35
> > 192.168.0.13:5984 192.168.0.8 admin GET
> >
> /stock/_design/company/_view/getallamexcompanies?reduce=false&skip=0&limit=101
> > 500 ok 1111
> > [error] 2020-06-09T00:33:11.129465Z couchdb@127.0.0.1 <0.1419.0>
> --------
> > gen_server <0.1419.0> terminated with reason: no match of right hand
> value
> > eof at couch_file:read_raw_iolist_int/3(line:627) <=
> > couch_file:handle_call/3(line:449) <=
> > gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647)
> > <= proc_lib:init_p_do_apply/3(line:247)
> > last msg: {pread_iolist,1190244510}
> > state:
> >
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7921>,106}},false,1767501252,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/80000000-9fffffff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"}]}]
> > extra: []
> > [error] 2020-06-09T00:33:11.132250Z couchdb@127.0.0.1 <0.1419.0>
> --------
> > CRASH REPORT Process (<0.1419.0>) with 1 neighbors exited with reason:
> no
> > match of right hand value eof at
> couch_file:read_raw_iolist_int/3(line:627)
> > <= couch_file:handle_call/3(line:449) <=
> > gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647)
> > <= proc_lib:init_p_do_apply/3(line:247) at
> gen_server:terminate/7(line:812)
> > <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> > {couch_file,init,['Argument__1']}, ancestors: [<0.1408.0>,<0.1407.0>],
> > messages: [], links: [<0.1408.0>], dictionary:
> >
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7921>,106}},"/dat..."}},...],
> > trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 573398
> > [error] 2020-06-09T00:33:11.132607Z couchdb@127.0.0.1 <0.1432.0>
> --------
> > gen_server <0.1432.0> terminated with reason: no match of right hand
> value
> > eof at couch_file:read_raw_iolist_int/3(line:627) <=
> > couch_file:handle_call/3(line:449) <=
> > gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647)
> > <= proc_lib:init_p_do_apply/3(line:247)
> > last msg:
> >
> {'EXIT',<0.1408.0>,{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
> > state: {st,<0.1408.0>,couch_mrview_index,undefined}
> > extra: []
> >
> >
> > error 2:
> > [error] 2020-06-09T00:40:25.580012Z couchdb@127.0.0.1 <0.3409.0>
> --------
> > gen_server <0.3409.0> terminated with reason: no match of right hand
> value
> > eof at couch_file:read_raw_iolist_int/3(line:627) <=
> > couch_file:handle_call/3(line:449) <=
> > gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647)
> > <= proc_lib:init_p_do_apply/3(line:247)
> > last msg: {pread_iolist,1223893408}
> > state:
> >
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7858>,108}},false,1686143327,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/00000000-1fffffff/stockanalysis.1584663342_design/mrview/98efc890b71fe5bf74611ad0868f39bf.view"}]}]
> > extra: []
> > [error] 2020-06-09T00:40:25.580132Z couchdb@127.0.0.1 <0.260.0> --------
> > Compaction daemon - an error ocurred while compacting the view group
> > `company` from database
> > `shards/00000000-1fffffff/stockanalysis.1584663342`:
> >
> {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
> > [error] 2020-06-09T00:40:25.580705Z couchdb@127.0.0.1 <0.3409.0>
> --------
> > CRASH REPORT Process (<0.3409.0>) with 1 neighbors exited with reason:
> no
> > match of right hand value eof at
> couch_file:read_raw_iolist_int/3(line:627)
> > <= couch_file:handle_call/3(line:449) <=
> > gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647)
> > <= proc_lib:init_p_do_apply/3(line:247) at
> gen_server:terminate/7(line:812)
> > <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> > {couch_file,init,['Argument__1']}, ancestors: [<0.3408.0>,<0.3407.0>],
> > messages: [], links: [<0.3408.0>], dictionary:
> >
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7858>,108}},"/dat..."}},...],
> > trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 627634
> > [error] 2020-06-09T00:40:25.581361Z couchdb@127.0.0.1 emulator --------
> > Error in process <0.3415.0> on node 'couchdb@127.0.0.1' with exit value:
> >
> {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
>
>
Re: couchdb file corruption
Posted by Jan Lehnardt <ja...@apache.org>.
Hi Sharath,
the error you are seeing originates in CouchDB’s file checksumming
feature[1]. When writing a block of data to disk, in either a
database or a view index, CouchDB also stores a checksum of the
block. When reading the block later, it checks if the stored
checksum and the newly read block still match. This is to prevent
hard drive errors sneaking into your database.
In your case, a block in a view index doesn’t match its checksum
anymore. The reasons for this could be manifold, but all of them
will have resulted in a bit or more on your storage device having
changed values behind CouchDB’s back.
To get back to working views, you can delete the index files and
rebuild them.
But more importantly:
1. do triple check what could have caused that block to go bad.
You don’t want to have this happen to your database files.
2. triple check your backups are up to date and restorable.
[1]: https://github.com/apache/couchdb/blob/3505281559513e2922484ebf0996a8846dcc0a34/src/couch/src/couch_file.erl#L173-L188
Best
Jan
—
> On 9. Jun 2020, at 02:44, Sharath <sh...@gmail.com> wrote:
>
> Hi,
>
> I'm encountering a file corruption in CouchDb 2.3.1 running on Ubuntu
> 18.04.1 LTS.
>
> The disk store is an ext4 SSD.
>
> I'm unable to access the view and couch logs shows the error below.
>
> Is there a way of knowing which view is corrupt (I have a few databases).
>
> I'm thinking of deleting all the views and recreating them - would that
> work?
>
> thanks!
>
> error 1:
> [emergency] 2020-06-09T00:33:10.122474Z couchdb@127.0.0.1 <0.1427.0>
> -------- File corruption in <0.951.0> at position 2668802729
> [error] 2020-06-09T00:33:10.123271Z couchdb@127.0.0.1 <0.1392.0> 9f5e486e35
> rexi_server: from: couchdb@127.0.0.1(<0.758.0>) mfa: fabric_rpc:map_view/5
> throw:{file_corruption,<<"file corruption">>}
> [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
> [error] 2020-06-09T00:33:10.123712Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
> req_err(4089041121) file_corruption : file corruption
> [<<"couch_mrview_util:get_view_index_state/5
> L137">>,<<"couch_mrview_util:get_view/4 L81">>,<<"couch_mrview:query_view/6
> L247">>,<<"rexi_server:init_p/3 L140">>]
> [notice] 2020-06-09T00:33:10.124065Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
> 192.168.0.13:5984 192.168.0.8 admin GET
> /stock/_design/company/_view/getallamexcompanies?reduce=false&skip=0&limit=101
> 500 ok 1111
> [error] 2020-06-09T00:33:11.129465Z couchdb@127.0.0.1 <0.1419.0> --------
> gen_server <0.1419.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
> last msg: {pread_iolist,1190244510}
> state:
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7921>,106}},false,1767501252,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/80000000-9fffffff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"}]}]
> extra: []
> [error] 2020-06-09T00:33:11.132250Z couchdb@127.0.0.1 <0.1419.0> --------
> CRASH REPORT Process (<0.1419.0>) with 1 neighbors exited with reason: no
> match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
> <= couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
> <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_file,init,['Argument__1']}, ancestors: [<0.1408.0>,<0.1407.0>],
> messages: [], links: [<0.1408.0>], dictionary:
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7921>,106}},"/dat..."}},...],
> trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> reductions: 573398
> [error] 2020-06-09T00:33:11.132607Z couchdb@127.0.0.1 <0.1432.0> --------
> gen_server <0.1432.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
> last msg:
> {'EXIT',<0.1408.0>,{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
> state: {st,<0.1408.0>,couch_mrview_index,undefined}
> extra: []
>
>
> error 2:
> [error] 2020-06-09T00:40:25.580012Z couchdb@127.0.0.1 <0.3409.0> --------
> gen_server <0.3409.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
> last msg: {pread_iolist,1223893408}
> state:
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7858>,108}},false,1686143327,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/00000000-1fffffff/stockanalysis.1584663342_design/mrview/98efc890b71fe5bf74611ad0868f39bf.view"}]}]
> extra: []
> [error] 2020-06-09T00:40:25.580132Z couchdb@127.0.0.1 <0.260.0> --------
> Compaction daemon - an error ocurred while compacting the view group
> `company` from database
> `shards/00000000-1fffffff/stockanalysis.1584663342`:
> {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
> [error] 2020-06-09T00:40:25.580705Z couchdb@127.0.0.1 <0.3409.0> --------
> CRASH REPORT Process (<0.3409.0>) with 1 neighbors exited with reason: no
> match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
> <= couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
> <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_file,init,['Argument__1']}, ancestors: [<0.3408.0>,<0.3407.0>],
> messages: [], links: [<0.3408.0>], dictionary:
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7858>,108}},"/dat..."}},...],
> trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> reductions: 627634
> [error] 2020-06-09T00:40:25.581361Z couchdb@127.0.0.1 emulator --------
> Error in process <0.3415.0> on node 'couchdb@127.0.0.1' with exit value:
> {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
Re: couchdb file corruption
Posted by Kiril Stankov <ki...@open-net.biz>.
Hi,
Couch (Erlang) log files are something only few chosen ones can
understand, and you will probably find someone on the list. I always
wondered why it shall be so hard to have a log line saying something like:
read past eof in file foo.bar. File seems to be corrupted.
IMHO, the problem is in
"/data/couchdb/.shards/80000000-9fffffff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"
which is a view file for the stock DB?
I guess that, yes, you can delete the view and try to recreate it.
Otherwise - try replicating the data to another instance, and then
delete the entire db, replicate it back and create the view again.
Best,
Kiril.
On 9.06.20 г. 3:44 ч., Sharath wrote:
> Hi,
>
> I'm encountering a file corruption in CouchDb 2.3.1 running on Ubuntu
> 18.04.1 LTS.
>
> The disk store is an ext4 SSD.
>
> I'm unable to access the view and couch logs shows the error below.
>
> Is there a way of knowing which view is corrupt (I have a few databases).
>
> I'm thinking of deleting all the views and recreating them - would that
> work?
>
> thanks!
>
> error 1:
> [emergency] 2020-06-09T00:33:10.122474Z couchdb@127.0.0.1 <0.1427.0>
> -------- File corruption in <0.951.0> at position 2668802729
> [error] 2020-06-09T00:33:10.123271Z couchdb@127.0.0.1 <0.1392.0> 9f5e486e35
> rexi_server: from: couchdb@127.0.0.1(<0.758.0>) mfa: fabric_rpc:map_view/5
> throw:{file_corruption,<<"file corruption">>}
> [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
> [error] 2020-06-09T00:33:10.123712Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
> req_err(4089041121) file_corruption : file corruption
> [<<"couch_mrview_util:get_view_index_state/5
> L137">>,<<"couch_mrview_util:get_view/4 L81">>,<<"couch_mrview:query_view/6
> L247">>,<<"rexi_server:init_p/3 L140">>]
> [notice] 2020-06-09T00:33:10.124065Z couchdb@127.0.0.1 <0.758.0> 9f5e486e35
> 192.168.0.13:5984 192.168.0.8 admin GET
> /stock/_design/company/_view/getallamexcompanies?reduce=false&skip=0&limit=101
> 500 ok 1111
> [error] 2020-06-09T00:33:11.129465Z couchdb@127.0.0.1 <0.1419.0> --------
> gen_server <0.1419.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
> last msg: {pread_iolist,1190244510}
> state:
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7921>,106}},false,1767501252,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/80000000-9fffffff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"}]}]
> extra: []
> [error] 2020-06-09T00:33:11.132250Z couchdb@127.0.0.1 <0.1419.0> --------
> CRASH REPORT Process (<0.1419.0>) with 1 neighbors exited with reason: no
> match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
> <= couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
> <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_file,init,['Argument__1']}, ancestors: [<0.1408.0>,<0.1407.0>],
> messages: [], links: [<0.1408.0>], dictionary:
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7921>,106}},"/dat..."}},...],
> trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> reductions: 573398
> [error] 2020-06-09T00:33:11.132607Z couchdb@127.0.0.1 <0.1432.0> --------
> gen_server <0.1432.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
> last msg:
> {'EXIT',<0.1408.0>,{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
> state: {st,<0.1408.0>,couch_mrview_index,undefined}
> extra: []
>
>
> error 2:
> [error] 2020-06-09T00:40:25.580012Z couchdb@127.0.0.1 <0.3409.0> --------
> gen_server <0.3409.0> terminated with reason: no match of right hand value
> eof at couch_file:read_raw_iolist_int/3(line:627) <=
> couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247)
> last msg: {pread_iolist,1223893408}
> state:
> [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7858>,108}},false,1686143327,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/00000000-1fffffff/stockanalysis.1584663342_design/mrview/98efc890b71fe5bf74611ad0868f39bf.view"}]}]
> extra: []
> [error] 2020-06-09T00:40:25.580132Z couchdb@127.0.0.1 <0.260.0> --------
> Compaction daemon - an error ocurred while compacting the view group
> `company` from database
> `shards/00000000-1fffffff/stockanalysis.1584663342`:
> {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
> [error] 2020-06-09T00:40:25.580705Z couchdb@127.0.0.1 <0.3409.0> --------
> CRASH REPORT Process (<0.3409.0>) with 1 neighbors exited with reason: no
> match of right hand value eof at couch_file:read_raw_iolist_int/3(line:627)
> <= couch_file:handle_call/3(line:449) <=
> gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
> <= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812)
> <= proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_file,init,['Argument__1']}, ancestors: [<0.3408.0>,<0.3407.0>],
> messages: [], links: [<0.3408.0>], dictionary:
> [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7858>,108}},"/dat..."}},...],
> trap_exit: false, status: running, heap_size: 6772, stack_size: 27,
> reductions: 627634
> [error] 2020-06-09T00:40:25.581361Z couchdb@127.0.0.1 emulator --------
> Error in process <0.3415.0> on node 'couchdb@127.0.0.1' with exit value:
> {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]}
>