[GitHub] [couchdb-couch-dbupdates] nickva commented on issue #3: background deletion for soft-deleted database

[GitBox <gi...@apache.org>]

nickva commented on issue #3:
URL: https://github.com/apache/couchdb-couch-dbupdates/pull/3#issuecomment-617470394


   @jiangphcn looks like a good start!
   
   I was think if we could perhaps just add this to fabric (fabric2*_*) modules like we did with indexing, also since soft deletion happens there, deletion logic would not be out of place there either.
   
   For the general structure what do you think about starting with something simpler at first, and say have only a singleton job, basically with type = <<"dbdelete">> and jobid <<"dbdelete_job">>. 
   
   Then we won't even need a couch_dbdelete_server. We just call a `couch_jobs:set_type_timeout(?DB_DELETE_JOB_TYPE, 6)` and `couch_jobs:add(
    undefined, ?DB_DELETE_JOB_TYPE, ?DB_DELETE_JOB, #{})` in some `init` function called by a supervisor. That would ensure this job would exist if it doesn't already.
   
   Then the `couch_dbdelete_worker` gen_server would wait to accept that job run it. Because of the locking we would know only one job would run in the whole cluster.
   
   In that worker we could run `fabric2_db:list_deleted_dbs_info(...)` but, let's use a callback there and accumulate batches of dbs, say 50 or 100 at a time. And for each db in the batch, we check the time limit and delete the old db instance. Here you can use a fabric2_util:pmap as I don't think we have a transactional interface to open and delete? Or just do a simple foreach loop at first.
   
   After we are done, the job can reschedule itself to run in at some future point in time say after 1 hour or so and then finish. To make the scheduling work in the accept function we'd use the `max_sched_time` of now + say 1 minute. 
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org