[GitHub] davisp commented on issue #1349: ddoc_cache_remove_test ets assertion fail on Jenkins

[GitBox <gi...@apache.org>]

davisp commented on issue #1349: ddoc_cache_remove_test ets assertion fail on Jenkins
URL: https://github.com/apache/couchdb/issues/1349#issuecomment-393224635
 
 
   Here's the relevant logs for the failure:
   
   ```
   [info] 2018-05-28T23:56:00.595674Z nonode@nohost <0.31.0> -------- Application ddoc_cache started on node nonode@nohost
   [error] 2018-05-28T23:56:00.688798Z nonode@nohost <0.6922.4> -------- ddoc_cache_ev :: started :: {ddoc_cache_entry_custom,{<<"eunit-test-db-fc3df6c7b8dd77eee5f87a58ab269484">>,ddoc_cache_remove_test}}
   [error] 2018-05-28T23:56:00.689891Z nonode@nohost <0.6922.4> -------- ddoc_cache_ev :: inserted :: {ddoc_cache_entry_custom,{<<"eunit-test-db-fc3df6c7b8dd77eee5f87a58ab269484">>,ddoc_cache_remove_test}}
   [error] 2018-05-28T23:56:00.756390Z nonode@nohost <0.6922.4> -------- ddoc_cache_ev :: removed :: {ddoc_cache_entry_custom,{<<"eunit-test-db-fc3df6c7b8dd77eee5f87a58ab269484">>,ddoc_cache_remove_test}}
   [error] 2018-05-28T23:56:00.756867Z nonode@nohost <0.6938.4> -------- ddoc_cache_ev :: started :: {ddoc_cache_entry_custom,{<<"eunit-test-db-fc3df6c7b8dd77eee5f87a58ab269484">>,ddoc_cache_remove_test}}
   [error] 2018-05-28T23:56:00.757463Z nonode@nohost <0.6938.4> -------- ddoc_cache_ev :: removed :: {ddoc_cache_entry_custom,{<<"eunit-test-db-fc3df6c7b8dd77eee5f87a58ab269484">>,ddoc_cache_remove_test}}
   [error] 2018-05-28T23:56:00.758095Z nonode@nohost <0.6942.4> -------- ddoc_cache_ev :: lru_init :: nil
   [error] 2018-05-28T23:56:00.758103Z nonode@nohost <0.6913.4> -------- gen_server ddoc_cache_lru terminated with reason: no match of right hand value not_found at ddoc_cache_lru:remove_key/2(line:307) <= ddoc_cache_lru:handle_info/2(line:218) <= gen_server:try_dispatch/4(line:601) <= gen_server:handle_msg/5(line:667) <= proc_lib:init_p_do_apply/3(line:247)
     last msg: {'EXIT',<0.6938.4>,normal}
        state: {st,<<>>,<<>>,<0.6914.4>}
   [error] 2018-05-28T23:56:00.758791Z nonode@nohost <0.6913.4> -------- CRASH REPORT Process ddoc_cache_lru (<0.6913.4>) with 0 neighbors exited with reason: no match of right hand value not_found at ddoc_cache_lru:remove_key/2(line:307) <= ddoc_cache_lru:handle_info/2(line:218) <= gen_server:try_dispatch/4(line:601) <= gen_server:handle_msg/5(line:667) <= proc_lib:init_p_do_apply/3(line:247) at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {ddoc_cache_lru,init,['Argument__1']}, ancestors: [ddoc_cache_sup,<0.6911.4>], messages: [{'EXIT',<0.6914.4>,killed}], links: [<0.6912.4>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 1423
   [error] 2018-05-28T23:56:00.759276Z nonode@nohost <0.6912.4> -------- Supervisor ddoc_cache_sup had child ddoc_cache_lru started with ddoc_cache_lru:start_link() at <0.6913.4> exit with reason no match of right hand value not_found at ddoc_cache_lru:remove_key/2(line:307) <= ddoc_cache_lru:handle_info/2(line:218) <= gen_server:try_dispatch/4(line:601) <= gen_server:handle_msg/5(line:667) <= proc_lib:init_p_do_apply/3(line:247) in context child_terminated
   ```
   
   For some reason when we're processing an exit message from an cache entry that failed to start we appear to be missing any record of the related database [1,2].
   
   [1] https://github.com/apache/couchdb/blob/master/src/ddoc_cache/src/ddoc_cache_lru.erl#L218
   [2] https://github.com/apache/couchdb/blob/master/src/ddoc_cache/src/ddoc_cache_lru.erl#L307
   
   A temp PR to try and narrow that down might be to add ?EVENT hooks around modifying those khashes which would theoretically show where timing is presumably getting out of whack from the normal expected case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services