Strange problems when PUTting to _config

[eric casteleijn <th...@gmail.com>]

This is on a snapshot of the 0.10 branch and may be fixed already. Since 
we're seeing this on a production server and not our development 
environments, it's hard to write a failing test.

After a lot of successful (after a fashion[*]) PUTting to and DELETEing 
from _config, we're seeing a sudden error in the logs, after which all 
subsequent ones seem to fail. The first failure looks like this 
(potential security sensitive info *****ed, but looks similar to earlier 
successful attempts):


[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.13623.4>] 'PUT' 
/_config/admins/50415 {1,1}
Headers: [{'Accept',"application/json"},
           {'Accept-Encoding',"identity"},
           {'Authorization',"OAuth realm=\"\", oauth_nonce=\"*****\", 
oauth_timestamp=\"1253842863\", oauth_consumer_key=\"*****\", 
oauth_signature_method=\"HMAC-SHA1\", oauth_version=\"1.0\", 
oauth_token=\"*****\", oauth_signature=\"*****\""},
           {'Content-Length',"10"},
           {'Host',"raspberry.canonical.com:9030"},
           {'User-Agent',"couchdb-python ?"}]

[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.13623.4>] OAuth Params: 
[{"oauth_nonce","03532062"},
                {"oauth_timestamp","1253842863"},
                {"oauth_consumer_key","*****"},
                {"oauth_signature_method","HMAC-SHA1"},
                {"oauth_version","1.0"},
                {"oauth_token","*****"},
                {"oauth_signature","*****"}]

[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.44.0>] saving to file 
'/etc/couchdb/local.ini', Config: '{{"admins","50415"},
                                                    "password"}'

[Fri, 25 Sep 2009 01:41:08 GMT] [error] [<0.13623.4>] Uncaught error in 
HTTP request: {exit,
                                  {timeout,
                                   {gen_server,call,
                                    [couch_config,
 
{set,"admins","50415","password",true}]}}}

[Fri, 25 Sep 2009 01:41:08 GMT] [info] [<0.13623.4>] Stacktrace: 
[{gen_server,call,2},
              {couch_httpd_misc_handlers,handle_config_req,1},
              {couch_httpd,handle_request,5},
              {mochiweb_http,headers,5},
              {proc_lib,init_p_do_apply,3}]

[Fri, 25 Sep 2009 01:41:08 GMT] [debug] [<0.13623.4>] httpd 500 error 
response:
 
{"error":"timeout","reason":"{gen_server,call,[couch_config,{set,\"admins\",\"50415\",\"password\",true}]}"}


[Fri, 25 Sep 2009 01:41:08 GMT] [info] [<0.13623.4>] 91.189.90.36 - - 
'PUT' /_config/admins/50415 500

[Fri, 25 Sep 2009 01:41:08 GMT] [debug] [<0.44.0>] saving to file 
'/etc/couchdb/local.ini', Config: '{{"admins","50415"},
                                                    "-hashed-*****"}'

NOTE: We get back a 500 (why, I don't know) and after that, the hashed 
password is happily written to the file anyway?! (Note also: for 
succesful writes, the order is the same, at least in the log file, which 
seems wrong: it sends back a 200 OK and then attempts the write to the 
.ini file...)

After this, all PUTS to _config seem to fail. Unfortunately it appears 
that one of my colleagues in their haste to fix this problem, has 
removed the local.ini file and started with a fresh one, so it is 
impossible for me to determine what state it was in. (I have asked an 
admin who is allowed to look in those files, but there seems to be 
nothing strange there now, and the admin user that was being PUT does 
not appear there.)

[*] Succesful in that we have to keep trying until we succeed, sometimes 
several hundred times. I think this is because some erlang bits need to 
be restarted after each write, like the auth_handler? Perhaps that is 
what failed to come back up at all after above mishap...

I've been seeing errors like this which seem to suggest that, but I suck 
at reading erlang/couchdb tracebacks:

[Fri, 25 Sep 2009 06:52:16 GMT] [error] [<0.3054.5>] {error_report,<0.24.0>,
     {<0.3054.5>,crash_report,
 
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
        {pid,<0.3054.5>},
        {registered_name,[]},
        {error_info,
            {exit,
                {timeout,
                    {gen_server,call,
                        [couch_config,
 
{get,"httpd","authentication_handlers",undefined}]}},
                [{gen_server,call,2},
                 {couch_httpd,handle_request,5},
                 {mochiweb_http,headers,5},
                 {proc_lib,init_p_do_apply,3}]}},
        {ancestors,
 
[couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
        {messages,[]},
        {links,[<0.63.0>,#Port<0.42195>]},
        {dictionary,[{jsonp,undefined}]},
        {trap_exit,false},
        {status,running},
        {heap_size,6765},
        {stack_size,24},
        {reductions,14822}],
       []]}}



-- 
- eric casteleijn
http://thisfred.blogspot.com