You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "rnewson (via GitHub)" <gi...@apache.org> on 2023/07/27 10:07:32 UTC
[GitHub] [couchdb] rnewson commented on pull request #4681: Introduce optional countermeasures as we run out of disk space
rnewson commented on PR #4681:
URL: https://github.com/apache/couchdb/pull/4681#issuecomment-1653309176
> It's a nice usability improvement and a good use of disk space monitor.
>
> There are a few comments and notes in-line.
>
> The one tricky aspect I think is how this interacts with the internal replicator. If one node is becoming full, we'd return an ` insufficient_storage` error from workers there. The request overall might even skip over that error and return a 201/202 if other shard copies reply with a success. That's a good behavior on one hand, on the other hand, there a good chance updates we blocked interactively would still make it to the affected nodes via the internal replicator and would continue filling up the disk.
>
> Not sure what a good solution might be. These are a few random thoughts I had:
>
> 1. We could crash the whole update_doc request if any shard copy is almost full? But even then, it might be too late as the successful requests on other nodes would still propagate via the internal replicator to the full node.
>
> 2. Is there a graceful way to stop internal replicator jobs and avoid them going into a crash loop and making a mess to avoid replicating data to nodes which are getting full?
>
> 3. I am wondering if at least for doc writes it would make sense to keep track of cluster wide disk usage (via a gossip between `couch_disk_monitor` gen_server or something like that) and return an immediate error at the fabric level without attempting any worker doc writes. That would increase the complexity of disk monitor, but would avoid the internal replicator loophole issue?
I think I will revise the block_database_writes() to be only _interactive_ writes. replicated writes will be permitted, as anti-entropy is critical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org