You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2022/10/24 20:14:11 UTC
[GitHub] [couchdb] nickva opened a new issue, #4239: Fix cleanup_index_files in smoosh so we can enable it by default
nickva opened a new issue, #4239:
URL: https://github.com/apache/couchdb/issues/4239
`cleanup_index_files` is currently disabled by default: https://github.com/apache/couchdb/blob/ebbcc7ec2a5bb6abf65738ff03f1be9f1be6017c/src/smoosh/src/smoosh_channel.erl#L569-L575
When enabled it can misbehave, especially in cases when there are lot of concurrent view compactions. As implemented currently `cleanup_index_files` spawns a simple, unmonitored process on each shard compaction start https://github.com/apache/couchdb/blob/ebbcc7ec2a5bb6abf65738ff03f1be9f1be6017c/src/smoosh/src/smoosh_channel.erl#L498-L501 :
During cleanup it performs a cluster-wide fabric call to fetch all design docs and then does wildcard view index folder search for stale files. Since this happens at the cluster database level we end performing the exact same cleanup action for the same fabric db Q times. With the fetching all ddocs and the wildcard directory search this operation can quickly overwhelm the cluster, falling behind while creating these unbounded processes, often doing redundant work.
At the same time, it would be nice to have the ability to cleanup stale view index files so they don't keep piling up and being able to default this setting to true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva closed issue #4239: Fix cleanup_index_files in smoosh so we can enable it by default
Posted by GitBox <gi...@apache.org>.
nickva closed issue #4239: Fix cleanup_index_files in smoosh so we can enable it by default
URL: https://github.com/apache/couchdb/issues/4239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #4239: Fix cleanup_index_files in smoosh so we can enable it by default
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #4239:
URL: https://github.com/apache/couchdb/issues/4239#issuecomment-1320233885
Index cleanup was improved in https://github.com/apache/couchdb/pull/4261
It's set to happen automatically in https://github.com/apache/couchdb/pull/4266 pending review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #4239: Fix cleanup_index_files in smoosh so we can enable it by default
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #4239:
URL: https://github.com/apache/couchdb/issues/4239#issuecomment-1289565739
Another idea is in addition to the index cleanup also remove stale view purge checkpoints (_local docs). During often view updates those can end up left being left, and when that happens they can prevent further purge request processing. This manifests as warnings and error in the logs as seen in some recent issues like https://github.com/apache/couchdb/issues/4181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #4239: Fix cleanup_index_files in smoosh so we can enable it by default
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #4239:
URL: https://github.com/apache/couchdb/issues/4239#issuecomment-1289559799
One idea how implement it is to add a separate cleanup channel to smoosh so instead spawning unmonitored processes there is channel which controls both capacity and concurrency of these cleanups. The database object would be clustered databases so if we're already enqueued clustered db `db1` once when we try to enqueue it again we ignore (de-duplicate) the second request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #4239: Fix cleanup_index_files in smoosh so we can enable it by default
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #4239:
URL: https://github.com/apache/couchdb/issues/4239#issuecomment-1320579445
With https://github.com/apache/couchdb/pull/4266 index cleanup got its won smoosh channel and is enabled by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org