You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2026/04/28 03:04:44 UTC

[PR] Fix cluster index and process cleanup [couchdb]

nickva opened a new pull request, #5986:
URL: https://github.com/apache/couchdb/pull/5986

   Previously, as described in #5980 we didn't perform a thorough index cleanup when ddocs changed. We only cleaned up on nodes where the design docs were located. That was true for a n=3 db and an n=3, db but may not be true in general in a cluster.
   
   To fix the issue, run a small gen_server responsible performing cluster index cleanup. To avoid spawning Q*N jobs, deduplicate the requests by delaying for up to 30 seconds per clustered db. For cleanup reuse and call the already existing fabric index file cleanup machinery. That accomplishes two things:
   
    - Starts a quicker index file cleanup. Previously we only did this during smoosh compaction runs. The view files could linger for a while until compaction in smoosh would be triggered.
   
    - Cleaning search index files also stops indexes on their (Java) side, so index file clean-up does "double duty" so speak when it comes to index shut down.
   
   Fix https://github.com/apache/couchdb/issues/5980
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#issuecomment-4336595452

   > > revert to the previous ddoc content and pick up those index files in disk again
   > 
   > it feels like we should make this an explicit feature rather than making an incident of the implementation. It might be too big a scope for this PR, but I think we should consider it going forward.
   > 
   > Something along the lines of renaming the file to include the deleted timestamp in the filename so we can purge it later (or rely on last modified times, if they are reliable), and scoop these files up before starting a fresh index build
   
   Oh that could be interesting. If we open an index we could look through the deleted views files to see if we have something recent (time and/or update seq) we can re-create from


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#discussion_r3155179524


##########
src/couch_index/src/couch_index_cleanup.erl:
##########
@@ -0,0 +1,98 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_cleanup).
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    schedule/1
+]).
+
+-export([
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+-export([
+    handle_db_event/3
+]).
+
+-define(DEFAULT_DELAY_MSEC, 30000).
+
+-record(st, {
+    pending = #{} :: #{binary() => reference()}
+}).
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+schedule(DbName) when is_binary(DbName) ->
+    gen_server:cast(?MODULE, {schedule, DbName, fanout}).
+
+init([]) ->
+    {ok, _} = couch_event:link_listener(?MODULE, handle_db_event, nil, [all_dbs]),
+    {ok, #st{}}.
+
+handle_call(Msg, _From, #st{} = St) ->
+    {stop, {invalid_call, Msg}, {invalid_call, Msg}, St}.
+
+handle_cast({schedule, DbName, Mode}, #st{pending = Pending} = St) ->
+    case maps:is_key(DbName, Pending) of
+        true ->
+            {noreply, St};
+        false ->
+            case Mode of
+                fanout -> fanout(DbName);
+                no_fanout -> ok
+            end,
+            TRef = erlang:send_after(delay_msec(), self(), {run_cleanup, DbName}),

Review Comment:
   Yup, exactly. That's why I kept the cleanup in smoosh as well. It's idempotent process we can always run cleanup again and it should cleanup any left overs or just do nothing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#discussion_r3155179524


##########
src/couch_index/src/couch_index_cleanup.erl:
##########
@@ -0,0 +1,98 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_cleanup).
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    schedule/1
+]).
+
+-export([
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+-export([
+    handle_db_event/3
+]).
+
+-define(DEFAULT_DELAY_MSEC, 30000).
+
+-record(st, {
+    pending = #{} :: #{binary() => reference()}
+}).
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+schedule(DbName) when is_binary(DbName) ->
+    gen_server:cast(?MODULE, {schedule, DbName, fanout}).
+
+init([]) ->
+    {ok, _} = couch_event:link_listener(?MODULE, handle_db_event, nil, [all_dbs]),
+    {ok, #st{}}.
+
+handle_call(Msg, _From, #st{} = St) ->
+    {stop, {invalid_call, Msg}, {invalid_call, Msg}, St}.
+
+handle_cast({schedule, DbName, Mode}, #st{pending = Pending} = St) ->
+    case maps:is_key(DbName, Pending) of
+        true ->
+            {noreply, St};
+        false ->
+            case Mode of
+                fanout -> fanout(DbName);
+                no_fanout -> ok
+            end,
+            TRef = erlang:send_after(delay_msec(), self(), {run_cleanup, DbName}),

Review Comment:
   Yup, exactly. That's why I kept the cleanup in smoosh as well. It's an idempotent process we can always run cleanup again and it should cleanup any left overs or just do nothing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva merged PR #5986:
URL: https://github.com/apache/couchdb/pull/5986


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "janl (via GitHub)" <gi...@apache.org>.
janl commented on code in PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#discussion_r3154939424


##########
src/couch_index/src/couch_index_cleanup.erl:
##########
@@ -0,0 +1,98 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_cleanup).
+-behaviour(gen_server).
+
+-export([
+    start_link/0,
+    schedule/1
+]).
+
+-export([
+    init/1,
+    handle_call/3,
+    handle_cast/2,
+    handle_info/2
+]).
+
+-export([
+    handle_db_event/3
+]).
+
+-define(DEFAULT_DELAY_MSEC, 30000).
+
+-record(st, {
+    pending = #{} :: #{binary() => reference()}
+}).
+
+start_link() ->
+    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
+
+schedule(DbName) when is_binary(DbName) ->
+    gen_server:cast(?MODULE, {schedule, DbName, fanout}).
+
+init([]) ->
+    {ok, _} = couch_event:link_listener(?MODULE, handle_db_event, nil, [all_dbs]),
+    {ok, #st{}}.
+
+handle_call(Msg, _From, #st{} = St) ->
+    {stop, {invalid_call, Msg}, {invalid_call, Msg}, St}.
+
+handle_cast({schedule, DbName, Mode}, #st{pending = Pending} = St) ->
+    case maps:is_key(DbName, Pending) of
+        true ->
+            {noreply, St};
+        false ->
+            case Mode of
+                fanout -> fanout(DbName);
+                no_fanout -> ok
+            end,
+            TRef = erlang:send_after(delay_msec(), self(), {run_cleanup, DbName}),

Review Comment:
   Just validating my mental model here. If we crash between this and the actual execution of `run_cleanup`, on restart, that old indexer processes are obviously gone and the index files get cleaned up when smoosh gets to it next time. So the only consequence is that index files will be deleted later than they would without this crash?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#discussion_r3193003095


##########
src/couch_index/src/couch_index_server.erl:
##########
@@ -358,51 +361,9 @@ handle_db_event(DbName, created, St) ->
 handle_db_event(DbName, deleted, St) ->
     gen_server:cast(St#st.server_name, {reset_indexes, DbName}),
     {ok, St};
-handle_db_event(<<"shards/", _/binary>> = DbName, {ddoc_updated, DDocId}, St) ->
-    %% this handle_db_event function must not crash (or it takes down the couch_index_server)
-    try
-        DDocResult = couch_util:with_db(DbName, fun(Db) ->
-            couch_db:open_doc(Db, DDocId, [ejson_body, ?ADMIN_CTX])
-        end),
-        LocalShards = mem3:local_shards(mem3:dbname(DbName)),
-        DbShards = [mem3:name(Sh) || Sh <- LocalShards],
-        lists:foreach(
-            fun(DbShard) ->
-                lists:foreach(
-                    fun({_DbShard, {_DDocId, Sig}}) ->
-                        % check if there are other ddocs with the same Sig for the same db
-                        SigDDocs = ets:match_object(St#st.by_db, {DbShard, {'$1', Sig}}),
-                        if
-                            length(SigDDocs) > 1 ->
-                                % remove records from by_db for this DDoc
-                                Args = [DbShard, DDocId, Sig],
-                                gen_server:cast(St#st.server_name, {rem_from_ets, Args});
-                            true ->
-                                % single DDoc with this Sig - close couch_index processes
-                                case ets:lookup(St#st.by_sig, {DbShard, Sig}) of
-                                    [{_, IndexPid}] ->
-                                        (catch gen_server:cast(
-                                            IndexPid, {ddoc_updated, DDocResult}
-                                        ));
-                                    [] ->
-                                        []
-                                end
-                        end
-                    end,
-                    ets:match_object(St#st.by_db, {DbShard, {DDocId, '$1'}})
-                )
-            end,
-            DbShards
-        ),
-        {ok, St}
-    catch
-        Class:Reason:Stack ->
-            couch_log:warning("~p: handle_db_event ~p for db ~p, reason ~p, stack ~p", [
-                ?MODULE, Class, DbName, Reason, Stack
-            ]),
-            gen_server:cast(St#st.server_name, {rem_from_ets, [DbName, Reason]}),

Review Comment:
   Good question! That's to skip over the clustered db shard names (they start with `<<"shards/...">>`) since we're handling them somewhere else now, but we still want handle the node-local dbs (the next clause) like we did before. If remove the `<<"shards/...">>` we'll end up handling both the clustered and local dbs in the same way which would be a new behavior from before



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#issuecomment-4332609139

   @willholley yeah this would shorten that time. However, when compaction would run is unpredictable in general, so we shouldn't rely on that feature. Previously, a design doc update could "kick" smoosh which could start an immediate cleanup and delete the files. Also, the configure file deletion options still applies (before and with this change) so if that's setup, the view files can still be recoverable with some file moves and copies perhaps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#discussion_r3192597045


##########
src/couch_index/src/couch_index_server.erl:
##########
@@ -358,51 +361,9 @@ handle_db_event(DbName, created, St) ->
 handle_db_event(DbName, deleted, St) ->
     gen_server:cast(St#st.server_name, {reset_indexes, DbName}),
     {ok, St};
-handle_db_event(<<"shards/", _/binary>> = DbName, {ddoc_updated, DDocId}, St) ->
-    %% this handle_db_event function must not crash (or it takes down the couch_index_server)
-    try
-        DDocResult = couch_util:with_db(DbName, fun(Db) ->
-            couch_db:open_doc(Db, DDocId, [ejson_body, ?ADMIN_CTX])
-        end),
-        LocalShards = mem3:local_shards(mem3:dbname(DbName)),
-        DbShards = [mem3:name(Sh) || Sh <- LocalShards],
-        lists:foreach(
-            fun(DbShard) ->
-                lists:foreach(
-                    fun({_DbShard, {_DDocId, Sig}}) ->
-                        % check if there are other ddocs with the same Sig for the same db
-                        SigDDocs = ets:match_object(St#st.by_db, {DbShard, {'$1', Sig}}),
-                        if
-                            length(SigDDocs) > 1 ->
-                                % remove records from by_db for this DDoc
-                                Args = [DbShard, DDocId, Sig],
-                                gen_server:cast(St#st.server_name, {rem_from_ets, Args});
-                            true ->
-                                % single DDoc with this Sig - close couch_index processes
-                                case ets:lookup(St#st.by_sig, {DbShard, Sig}) of
-                                    [{_, IndexPid}] ->
-                                        (catch gen_server:cast(
-                                            IndexPid, {ddoc_updated, DDocResult}
-                                        ));
-                                    [] ->
-                                        []
-                                end
-                        end
-                    end,
-                    ets:match_object(St#st.by_db, {DbShard, {DDocId, '$1'}})
-                )
-            end,
-            DbShards
-        ),
-        {ok, St}
-    catch
-        Class:Reason:Stack ->
-            couch_log:warning("~p: handle_db_event ~p for db ~p, reason ~p, stack ~p", [
-                ?MODULE, Class, DbName, Reason, Stack
-            ]),
-            gen_server:cast(St#st.server_name, {rem_from_ets, [DbName, Reason]}),

Review Comment:
   After removing `handle_db_event(<<"shards/", _/binary>> = DbName, {ddoc_updated, DDocId}, St) ->`.
   
   > `handle_cast({rem_from_ets, [DbName, Reason]}, State) ->`
   
   Do we still need this `handle_cast()` ^? Is it retained for backward compatibility?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on code in PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#discussion_r3193003095


##########
src/couch_index/src/couch_index_server.erl:
##########
@@ -358,51 +361,9 @@ handle_db_event(DbName, created, St) ->
 handle_db_event(DbName, deleted, St) ->
     gen_server:cast(St#st.server_name, {reset_indexes, DbName}),
     {ok, St};
-handle_db_event(<<"shards/", _/binary>> = DbName, {ddoc_updated, DDocId}, St) ->
-    %% this handle_db_event function must not crash (or it takes down the couch_index_server)
-    try
-        DDocResult = couch_util:with_db(DbName, fun(Db) ->
-            couch_db:open_doc(Db, DDocId, [ejson_body, ?ADMIN_CTX])
-        end),
-        LocalShards = mem3:local_shards(mem3:dbname(DbName)),
-        DbShards = [mem3:name(Sh) || Sh <- LocalShards],
-        lists:foreach(
-            fun(DbShard) ->
-                lists:foreach(
-                    fun({_DbShard, {_DDocId, Sig}}) ->
-                        % check if there are other ddocs with the same Sig for the same db
-                        SigDDocs = ets:match_object(St#st.by_db, {DbShard, {'$1', Sig}}),
-                        if
-                            length(SigDDocs) > 1 ->
-                                % remove records from by_db for this DDoc
-                                Args = [DbShard, DDocId, Sig],
-                                gen_server:cast(St#st.server_name, {rem_from_ets, Args});
-                            true ->
-                                % single DDoc with this Sig - close couch_index processes
-                                case ets:lookup(St#st.by_sig, {DbShard, Sig}) of
-                                    [{_, IndexPid}] ->
-                                        (catch gen_server:cast(
-                                            IndexPid, {ddoc_updated, DDocResult}
-                                        ));
-                                    [] ->
-                                        []
-                                end
-                        end
-                    end,
-                    ets:match_object(St#st.by_db, {DbShard, {DDocId, '$1'}})
-                )
-            end,
-            DbShards
-        ),
-        {ok, St}
-    catch
-        Class:Reason:Stack ->
-            couch_log:warning("~p: handle_db_event ~p for db ~p, reason ~p, stack ~p", [
-                ?MODULE, Class, DbName, Reason, Stack
-            ]),
-            gen_server:cast(St#st.server_name, {rem_from_ets, [DbName, Reason]}),

Review Comment:
   Good question! That's to skip over the clustered db shard names (they start with `<<"shards/...">>`) since we're handling them somewhere else now, but we still want handle the node-local dbs (the next clause) like we did before. If remove the `<<"shards/...">>` we'll end up handling both the clustered and local dbs in the same which would be a new behavior from before



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "janl (via GitHub)" <gi...@apache.org>.
janl commented on PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#issuecomment-4336207810

   > revert to the previous ddoc content and pick up those index files in disk again
   
   it feels like we should make this an explicit feature rather than making an incident of the implementation. It might be too big a scope for this PR, but I think we should consider it going forward.
   
   Something along the lines of renaming the file to include the deleted timestamp in the filename so we can purge it later (or rely on last modified times, if they are reliable), and scoop these files up before starting a fresh index build


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "willholley (via GitHub)" <gi...@apache.org>.
willholley commented on PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#issuecomment-4333030146

   @nickva yep - not a blocker from me, just an operational change we'll need to be aware of.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix cluster index and process cleanup [couchdb]

Posted by "willholley (via GitHub)" <gi...@apache.org>.
willholley commented on PR #5986:
URL: https://github.com/apache/couchdb/pull/5986#issuecomment-4332231605

   Does this change mean that users who inadvertently trigger a full index rebuild by modifying a design document will have a much smaller window during which to "undo" the change (revert to the previous ddoc content and pick up those index files in disk again)? We've definitely used the delay in index cleanup to recover from that kind of situation, particularly when an index takes hours/days to rebuild.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org