You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Antony Blakey (JIRA)" <ji...@apache.org> on 2009/05/28 23:51:45 UTC
[jira] Created: (COUCHDB-365) Replication hangs and won't restart
Replication hangs and won't restart
-----------------------------------
Key: COUCHDB-365
URL: https://issues.apache.org/jira/browse/COUCHDB-365
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 0.9.1
Environment: Replicate to Ubuntu from OSX, both running R13B
Reporter: Antony Blakey
Partway through a replication I get this:
{"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
as the response from the replication request. Attempting to restart replication results in:
[info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-365) Replication hangs and won't restart
Posted by "Antony Blakey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714200#action_12714200 ]
Antony Blakey commented on COUCHDB-365:
---------------------------------------
And after a few of those, I get this error after a restart, which cannot be proceeded beyond without further restarts of couchdb. It seems random whether it will continue on to the previous error, or immediately crapout with this one.
[debug] [<0.125.0>] 'POST' /_replicate {1,1}
Headers: [{'Accept',"*/*"},
{'Content-Length',"102"},
{'Content-Type',"application/x-www-form-urlencoded"},
{'Host',"localhost:5984"},
{'User-Agent',"curl/7.18.0 (i486-pc-linux-gnu) libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.1"}]
[debug] [<0.137.0>] couch_rep HTTP get request: http://localhost:5985/acumen-curricula/
[error] [<0.137.0>] {error_report,<0.24.0>,
{<0.137.0>,crash_report,
[[{initial_call,{couch_rep,init,['Argument__1']}},
{pid,<0.137.0>},
{registered_name,[]},
{error_info,
{exit,
{badarg,
[{erlang,list_to_existing_atom,["db_name"]},
{couch_rep,'-get_db_info/1-lc$^0/1-0-',1},
{couch_rep,get_db_info,1},
{couch_rep,init,1},
{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]},
[{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_rep_sup,couch_primary_services,couch_server_sup,<0.1.0>]},
{messages,[]},
{links,[<0.42.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,679}],
[]]}}
[error] [<0.125.0>] Uncaught error in HTTP request: {error,
{case_clause,
{error,
{{badarg,
[{erlang,list_to_existing_atom,
["db_name"]},
{couch_rep,'-get_db_info/1-lc$^0/1-0-',
1},
{couch_rep,get_db_info,1},
{couch_rep,init,1},
{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]},
{child,undefined,
"1f78d088910dc73016c0b16b2fb84aac",
{gen_server,start_link,
[couch_rep,
["1f78d088910dc73016c0b16b2fb84aac",
{remote,
<<"http://localhost:5985/acumen-curricula/">>,
[]},
{remote,
<<"http://localhost:5984/acumen-curricula/">>,
[]}],
[]]},
transient,1,worker,
[couch_rep]}}}}}
[info] [<0.125.0>] Stacktrace: [{couch_rep,replicate,2},
{couch_httpd_misc_handlers,handle_replicate_req,1},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[debug] [<0.125.0>] httpd 500 error response:
{"error":"case_clause","reason":"{error,{{badarg,[{erlang,list_to_existing_atom,[\"db_name\"]},\n {couch_rep,'-get_db_info/1-lc$^0/1-0-',1},\n {couch_rep,get_db_info,1},\n {couch_rep,init,1},\n {gen_server,init_it,6},\n {proc_lib,init_p_do_apply,3}]},\n {child,undefined,\"1f78d088910dc73016c0b16b2fb84aac\",\n {gen_server,start_link,\n [couch_rep,\n [\"1f78d088910dc73016c0b16b2fb84aac\",\n {remote,<<\"http://localhost:5985/acumen-curricula/\">>,\n []},\n {remote,<<\"http://localhost:5984/acumen-curricula/\">>,\n []}],\n []]},\n transient,1,worker,\n [couch_rep]}}}"}
[info] [<0.125.0>] 127.0.0.1 - - 'POST' /_replicate 500
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (COUCHDB-365) Replication hangs and won't restart
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski closed COUCHDB-365.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.10
We stopped using this code path anyway.
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
> Assignee: Adam Kocoloski
> Fix For: 0.10
>
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-365) Replication hangs and won't restart
Posted by "Antony Blakey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714294#action_12714294 ]
Antony Blakey commented on COUCHDB-365:
---------------------------------------
The original error is in this code:
should_flush(_DocCount) ->
MeAndMyLinks = [self()|
[P || P <- element(2,process_info(self(),links)), is_pid(P)]],
case length(MeAndMyLinks)/2 > ?BUFFER_NATTACHMENTS of
true -> true;
false ->
case memory_footprint(MeAndMyLinks) > 2*?BUFFER_MEMORY of
true ->
[garbage_collect(Pid) || Pid <- MeAndMyLinks],
memory_footprint(MeAndMyLinks) > ?BUFFER_MEMORY;
false -> false
end
end.
The problem being that the second time memory_footprint is called the elements of MeAndMyLinks may not be valid because of the preceding garbage_collect.
I fixed this by replacing that second call by this:
memory_footprint([self()| [P || P <- element(2,process_info(self(),links)), is_pid(P)]]) > ?BUFFER_MEMORY;
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-365) Replication hangs and won't restart
Posted by "Antony Blakey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714210#action_12714210 ]
Antony Blakey commented on COUCHDB-365:
---------------------------------------
The immediately preceding problem doesn't occur/goes away if I make a different request first, such as a futon request. If however I start couchdb and then make a replication request, it always fails with the above error,
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (COUCHDB-365) Replication hangs and won't restart
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan Lehnardt reassigned COUCHDB-365:
------------------------------------
Assignee: Adam Kocoloski
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
> Assignee: Adam Kocoloski
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-365) Replication hangs and won't restart
Posted by "Antony Blakey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714195#action_12714195 ]
Antony Blakey commented on COUCHDB-365:
---------------------------------------
After this happens, I get repeated hangs with the log showing this:
[debug] [<0.89.0>] Attachment URL http://localhost:5985/acumen-curricula/c16669078d5349503423e44095de25a3/Eng3MainIdea.pdf?rev=31-3046362463
[debug] [<0.89.0>] streaming attachment Status "200" Headers [{"Transfer-Encoding","chunked"},
{"Server",
"CouchDB/0.10.0a (Erlang OTP/R13B)"},
{"ETag","\"31-3046362463\""},
{"Date",
"Thu, 28 May 2009 22:11:11 GMT"},
{"Content-Type","application/pdf"},
{"Cache-Control",
"must-revalidate"}]
[debug] [<0.89.0>] ProcessMem 56738000 BinaryMem 24014824
[debug] [<0.89.0>] ProcessMem 32203508 BinaryMem 2156534
but each time at a different point. This may well be unrelated, but I'm not sure.
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-365) Replication hangs and won't restart
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714305#action_12714305 ]
Adam Kocoloski commented on COUCHDB-365:
----------------------------------------
Thanks, Antony. The second problem you encountered is due to the use of list_to_existing_atom/1. I replaced it with plain-old list_to_atom.
Instead of recalculating the "MeAndMyLinks" list every time, I added some guards to memory_footprint so that it doesn't try to call process_info with a dead Pid. Hopefully this resolves the issue.
> Replication hangs and won't restart
> -----------------------------------
>
> Key: COUCHDB-365
> URL: https://issues.apache.org/jira/browse/COUCHDB-365
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
> Affects Versions: 0.9.1
> Environment: Replicate to Ubuntu from OSX, both running R13B
> Reporter: Antony Blakey
>
> Partway through a replication I get this:
> {"error":"json_encode","reason":"{bad_term,{couch_rep,'-memory_footprint/1-fun-0-',2}}"}
> as the response from the replication request. Attempting to restart replication results in:
> [info] [<0.5511.0>] replication "1f78d088910dc73016c0b16b2fb84aac" already running at <0.4203.0>
> but the replication has hung.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.