You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2022/02/18 12:32:15 UTC

[GitHub] [couchdb] iilyak opened a new pull request #3933: Debug for sharded index server

iilyak opened a new pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933


   ## Overview
   
   Recent refactor of `couch_index_server` made it a sharded implementation. This means multiple processes are in play now. This made it somewhat harder to bounce in production in case of problems. Also the `couch_debug:print_linked_processes(couch_index_server)` had an assumption that `couch_index_server` is a single process.
   
   This PR does two very related things 
   
   1. fixes `couch_debug:print_linked_processes(couch_index_server)`
   2. adds `couch_index_debug` module with auxiliary functions to restart `couch_index_server` processes safely. 
   
   ## Testing recommendations
   
   1. compile, start, remsh
   2. `couch_debug:print_linked_processes(couch_index_server).`
   ```
   |name                                              | reductions | message_queue_len |    memory    |id
   |--------------------------------------------------|------------|-------------------|--------------|--
   |index_server_1[<0.320.0>]                         |    1127    |         0         |    17000     |
   |  couch_secondary_services[<0.312.0>]             |   93596    |         0         |    68632     |
   |  couch_event_listener:do_init/3[<0.323.0>]       |    203     |         0         |     2856     |
   |    index_server_1[<0.320.0>]                     |    1127    |         0         |    17000     |
   |                                                  |            |                   |              |
   |index_server_2[<0.324.0>]                         |    290     |         0         |     6088     |
   |  couch_secondary_services[<0.312.0>]             |   93598    |         0         |    68632     |
   |  couch_event_listener:do_init/3[<0.326.0>]       |    169     |         0         |     2856     |
   |    index_server_2[<0.324.0>]                     |    290     |         0         |     6088     |
   ....
   ```
   3. `couch_index_debug:help().`
   ```
   [names,print_linked_processes,busy,restart_busy,restart]
   ```
   4. `couch_index_debug:help(names)`
   5. `couch_index_debug:help(print_linked_processes)`
   6. `couch_index_debug:help(busy)`
   7. `couch_index_debug:help(restart_busy)`
   8. `couch_index_debug:help(restart)`
   9. Note pid of `whereis(index_server_8).` and do `couch_index_debug:restart(index_server_8)`
   
   ## Related Issues or Pull Requests
   
   - https://github.com/apache/couchdb/pull/3860
   
   ## Checklist
   
   - [x] Code is written and works correctly
   - [ ] Changes are covered by tests
   - [ ] Any new configurable parameters are documented in `rel/overlay/etc/default.ini`
   - [ ] A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810237724



##########
File path: src/couch/src/couch_debug.erl
##########
@@ -446,6 +553,57 @@ shorten_path(Path) ->
     <<_:Len/binary, Rest/binary>> = File,
     binary_to_list(Rest).
 
+-spec restart(Name :: process_name()) ->
+    Pid :: pid() | timeout.
+
+restart(Name) ->
+    Res = test_util:with_process_restart(Name, fun() ->

Review comment:
       *Note*: I am calling a function from `test_util` module. Is it worth to move the `with_process_restart` (and friends) into `couch_debug` and create a dispatch in `test_util`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810040656



##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,170 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    restart_busy/1,
+    restart_busy/2,
+    restart/1
+]).
+
+%% Use it as `-> throw({unknown_mode, Reason :: term()}).`
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy,
+        restart
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold.
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(Thereshold)
+    restart_busy(Thereshold, DelayInMsec)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold. Then it restart the identified processes.
+    The restarts happen sequentially with a given DelayInMsec between them.
+    If DelayInMsec is not provided the default value is one second.
+    The function doesn't proceed to next server until the replacement server
+    process starts.
+    ---
+    ", []);
+help(restart) ->
+    io:format("
+    restart(ServerName)
+    --------------
+
+    Restart a couch_index_server with given ServerName and wait for
+    replacement process to start.
+    ---
+    ", []);
+help(Unknown) ->
+    io:format("Unknown function: `~p`. Please try one of the following:~n", [Unknown]),
+    [io:format("    - ~s~n", [Function]) || Function <- help()],
+    io:format("    ---~n", []),
+    ok.
+
+-spec names() -> [process_name()].
+
+names() ->
+    couch_index_server:names().
+
+-spec print_linked_processes() -> ok.
+
+print_linked_processes() ->
+    couch_debug:print_linked_processes(couch_index_server).
+
+-spec busy(Thershold :: pos_integer()) ->
+    [Name :: process_name()].
+
+busy(Threshold) when Threshold > 0 ->
+    lists:filter(
+        fun(Name) ->
+            case (catch process_info(whereis(Name), message_queue_len)) of
+                QueueLen when is_integer(QueueLen) andalso QueueLen > Threshold ->
+                    true;
+                _ ->
+                    false
+            end
+        end,
+        names()
+    ).
+
+-spec restart_busy(Threshold :: pos_integer()) ->

Review comment:
       I think I should move it into couch_debug.erl for re-use.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810038076



##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,170 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    restart_busy/1,
+    restart_busy/2,
+    restart/1
+]).
+
+%% Use it as `-> throw({unknown_mode, Reason :: term()}).`
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy,
+        restart
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold.
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(Thereshold)
+    restart_busy(Thereshold, DelayInMsec)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold. Then it restart the identified processes.
+    The restarts happen sequentially with a given DelayInMsec between them.
+    If DelayInMsec is not provided the default value is one second.
+    The function doesn't proceed to next server until the replacement server
+    process starts.
+    ---
+    ", []);
+help(restart) ->
+    io:format("
+    restart(ServerName)
+    --------------
+
+    Restart a couch_index_server with given ServerName and wait for
+    replacement process to start.
+    ---
+    ", []);
+help(Unknown) ->
+    io:format("Unknown function: `~p`. Please try one of the following:~n", [Unknown]),
+    [io:format("    - ~s~n", [Function]) || Function <- help()],
+    io:format("    ---~n", []),
+    ok.
+
+-spec names() -> [process_name()].
+
+names() ->
+    couch_index_server:names().
+
+-spec print_linked_processes() -> ok.
+
+print_linked_processes() ->
+    couch_debug:print_linked_processes(couch_index_server).
+
+-spec busy(Thershold :: pos_integer()) ->

Review comment:
       This is not couch_index_server specific. Therefore I am considering to move it into `couch_debug.erl` for reuse.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810038076



##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,170 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    restart_busy/1,
+    restart_busy/2,
+    restart/1
+]).
+
+%% Use it as `-> throw({unknown_mode, Reason :: term()}).`
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy,
+        restart
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold.
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(Thereshold)
+    restart_busy(Thereshold, DelayInMsec)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold. Then it restart the identified processes.
+    The restarts happen sequentially with a given DelayInMsec between them.
+    If DelayInMsec is not provided the default value is one second.
+    The function doesn't proceed to next server until the replacement server
+    process starts.
+    ---
+    ", []);
+help(restart) ->
+    io:format("
+    restart(ServerName)
+    --------------
+
+    Restart a couch_index_server with given ServerName and wait for
+    replacement process to start.
+    ---
+    ", []);
+help(Unknown) ->
+    io:format("Unknown function: `~p`. Please try one of the following:~n", [Unknown]),
+    [io:format("    - ~s~n", [Function]) || Function <- help()],
+    io:format("    ---~n", []),
+    ok.
+
+-spec names() -> [process_name()].
+
+names() ->
+    couch_index_server:names().
+
+-spec print_linked_processes() -> ok.
+
+print_linked_processes() ->
+    couch_debug:print_linked_processes(couch_index_server).
+
+-spec busy(Thershold :: pos_integer()) ->

Review comment:
       This is not couch_index_server specific. Therefore I am considering to move it into `couch_debug.erl` for reuse. With a slight change in funuction signature. It would become `busy(Threshold, ProcessList) -> [proccess_name()].`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak merged pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak merged pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] jaydoane commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
jaydoane commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810567375



##########
File path: src/couch/src/couch_debug.erl
##########
@@ -202,6 +281,30 @@ help(Unknown) ->
     io:format("    ---~n", []),
     ok.
 
+-spec busy(ProcessList :: [process_name()], Thershold :: pos_integer()) ->
+    [Name :: process_name()].
+
+busy(ProcessList, Threshold) when Threshold > 0 ->
+    busy(ProcessList, Threshold, message_queue_len).
+
+-spec busy(
+    ProcessList :: [process_name()], Thershold :: pos_integer(), Property :: busy_properties()

Review comment:
       s/Thershold/Threshold

##########
File path: src/couch/src/couch_debug.erl
##########
@@ -446,6 +553,57 @@ shorten_path(Path) ->
     <<_:Len/binary, Rest/binary>> = File,
     binary_to_list(Rest).
 
+-spec restart(Name :: process_name()) ->
+    Pid :: pid() | timeout.
+
+restart(Name) ->
+    Res = test_util:with_process_restart(Name, fun() ->

Review comment:
       The main downside IMO to moving them is that `with_process_restart` uses the same sneaky trick with respect to time units as `test_util:wait` and friends, so keeping them together might be slightly less bewildering to someone unfamiliar with that code. I'm pretty neutral, though, and just wish all implementations were a bit less sneaky.

##########
File path: src/couch/src/couch_debug.erl
##########
@@ -45,11 +65,34 @@ help() ->
         map,
         fold,
         linked_processes_info,
-        print_linked_processes
+        print_linked_processes,
+        restart,
+        restart_busy
     ].
 
--spec help(Function :: atom()) -> ok.
+-spec help(Function :: function_name()) -> ok.
 %% erlfmt-ignore
+help(busy) ->
+    io:format("
+    busy(ProcessList, Threshold)
+    busy(ProcessList, Threshold, Property)
+    --------------
+
+    Iterate over given list of named processes and returns the ones with
+    a Property value greater than provided Threshold.
+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow

Review comment:
       s/bellow/below

##########
File path: src/couch/src/couch_debug.erl
##########
@@ -446,6 +553,57 @@ shorten_path(Path) ->
     <<_:Len/binary, Rest/binary>> = File,
     binary_to_list(Rest).
 
+-spec restart(Name :: process_name()) ->
+    Pid :: pid() | timeout.
+
+restart(Name) ->
+    Res = test_util:with_process_restart(Name, fun() ->
+        exit(whereis(Name), kill)
+    end),
+    case Res of
+        {Pid, true} ->
+            Pid;
+        timeout ->
+            timeout
+    end.
+
+-spec restart_busy(ProcessList :: [process_name()], Threshold :: pos_integer()) ->
+    throw({timeout, Name :: process_name()}).

Review comment:
       does this also return `ok`?
   ```
   (node1@127.0.0.1)28> couch_index_debug:restart_busy(4000, 50, reductions).
   ok
   ```

##########
File path: src/couch/src/couch_debug.erl
##########
@@ -202,6 +281,30 @@ help(Unknown) ->
     io:format("    ---~n", []),
     ok.
 
+-spec busy(ProcessList :: [process_name()], Thershold :: pos_integer()) ->

Review comment:
       s/Thershold/Threshold

##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,171 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    busy/2,
+    restart_busy/1,
+    restart_busy/2,
+    restart_busy/3
+]).
+
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    busy(Thereshold, Property)
+    --------------
+
+    Finds list of couch_index_server processes and returns the ones with
+    a Property value greater than provided Threshold.
+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow
+
+    - heap_size
+    - memory
+    - message_queue_len (default)
+    - reductions
+    - total_heap_size
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(Thereshold)
+    restart_busy(Thereshold, DelayInMsec)
+    restart_busy(Thereshold, DelayInMsec, Property)
+    --------------
+
+    Finds list of couch_index_server processes and returns the ones with
+    a Property value greater than provided Threshold.
+
+    Then it restart the identified processes.
+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow

Review comment:
       s/bellow/below

##########
File path: src/couch/src/couch_debug.erl
##########
@@ -91,6 +134,42 @@ help(process_name) ->
     - '$initial_call' key in process dictionary
     - process_info(Pid, initial_call)
 
+    ---
+    ", []);
+help(restart) ->
+    io:format("
+    restart(ServerName)
+    --------------
+
+    Restart a process with given ServerName and wait for
+    replacement process to start.
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(ProcessList, Thereshold)
+    restart_busy(ProcessList, Thereshold, DelayInMsec)
+    --------------
+
+    Iterate over given list of named processes and returns the ones with
+    a Property value greater than provided Threshold.
+
+    Then it restart the identified processes.
+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow

Review comment:
       s/bellow/below

##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,171 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    busy/2,
+    restart_busy/1,
+    restart_busy/2,
+    restart_busy/3
+]).
+
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    busy(Thereshold, Property)
+    --------------
+
+    Finds list of couch_index_server processes and returns the ones with
+    a Property value greater than provided Threshold.
+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow

Review comment:
       s/bellow/below




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810236330



##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,170 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    restart_busy/1,
+    restart_busy/2,
+    restart/1
+]).
+
+%% Use it as `-> throw({unknown_mode, Reason :: term()}).`
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy,
+        restart
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold.
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(Thereshold)
+    restart_busy(Thereshold, DelayInMsec)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold. Then it restart the identified processes.
+    The restarts happen sequentially with a given DelayInMsec between them.
+    If DelayInMsec is not provided the default value is one second.
+    The function doesn't proceed to next server until the replacement server
+    process starts.
+    ---
+    ", []);
+help(restart) ->
+    io:format("
+    restart(ServerName)
+    --------------
+
+    Restart a couch_index_server with given ServerName and wait for
+    replacement process to start.
+    ---
+    ", []);
+help(Unknown) ->
+    io:format("Unknown function: `~p`. Please try one of the following:~n", [Unknown]),
+    [io:format("    - ~s~n", [Function]) || Function <- help()],
+    io:format("    ---~n", []),
+    ok.
+
+-spec names() -> [process_name()].
+
+names() ->
+    couch_index_server:names().
+
+-spec print_linked_processes() -> ok.
+
+print_linked_processes() ->
+    couch_debug:print_linked_processes(couch_index_server).
+
+-spec busy(Thershold :: pos_integer()) ->
+    [Name :: process_name()].
+
+busy(Threshold) when Threshold > 0 ->
+    lists:filter(
+        fun(Name) ->
+            case (catch process_info(whereis(Name), message_queue_len)) of
+                QueueLen when is_integer(QueueLen) andalso QueueLen > Threshold ->
+                    true;
+                _ ->
+                    false
+            end
+        end,
+        names()
+    ).
+
+-spec restart_busy(Threshold :: pos_integer()) ->

Review comment:
       done in https://github.com/apache/couchdb/pull/3933/commits/8da3715c559095d8da5da8a07c838c4404c4fc7b




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] iilyak commented on a change in pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
iilyak commented on a change in pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#discussion_r810235853



##########
File path: src/couch_index/src/couch_index_debug.erl
##########
@@ -0,0 +1,170 @@
+% Licensed under the Apache License, Version 2.0 (the "License"); you may not
+% use this file except in compliance with the License. You may obtain a copy of
+% the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+% License for the specific language governing permissions and limitations under
+% the License.
+
+-module(couch_index_debug).
+
+-export([
+    help/0,
+    help/1
+]).
+
+-export([
+    names/0,
+    print_linked_processes/0,
+    busy/1,
+    restart_busy/1,
+    restart_busy/2,
+    restart/1
+]).
+
+%% Use it as `-> throw({unknown_mode, Reason :: term()}).`
+-type throw(_Reason) :: no_return().
+
+-type process_name() :: atom().
+-type function_name() :: atom().
+
+help() ->
+    [
+        %% list of provided commands
+        names,
+        print_linked_processes,
+        busy,
+        restart_busy,
+        restart
+    ].
+
+-spec help(Function :: function_name()) -> ok.
+%% erlfmt-ignore
+help(names) ->
+    io:format("
+    names()
+    --------------
+
+    Returns list of named processes which constitutes
+    a sharded couch_index_server
+    ---
+    ", []);
+help(print_linked_processes) ->
+    io:format("
+    print_linked_processes()
+    --------------
+
+    Print cluster of linked processes. The output would look like similar to:
+
+    |name                                              | reductions | message_queue_len |    memory    |id
+    |--------------------------------------------------|------------|-------------------|--------------|--
+    |index_server_1[<0.320.0>]                         |    1115    |         0         |    17000     |
+    |  couch_secondary_services[<0.312.0>]             |   93258    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.323.0>]       |    195     |         0         |     2856     |
+    |    index_server_1[<0.320.0>]                     |    1115    |         0         |    17000     |
+    |                                                  |            |                   |              |
+    |index_server_2[<0.324.0>]                         |    278     |         0         |     6088     |
+    |  couch_secondary_services[<0.312.0>]             |   93260    |         0         |    68600     |
+    |  couch_event_listener:do_init/3[<0.326.0>]       |    161     |         0         |     2856     |
+    |    index_server_2[<0.324.0>]                     |    278     |         0         |     6088     |
+    ---
+    ", []);
+help(busy) ->
+    io:format("
+    busy(Thereshold)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold.
+    ---
+    ", []);
+help(restart_busy) ->
+    io:format("
+    restart_busy(Thereshold)
+    restart_busy(Thereshold, DelayInMsec)
+    --------------
+
+    Finds list of couch_index_server processes with message box size greater
+    than provided Threshold. Then it restart the identified processes.
+    The restarts happen sequentially with a given DelayInMsec between them.
+    If DelayInMsec is not provided the default value is one second.
+    The function doesn't proceed to next server until the replacement server
+    process starts.
+    ---
+    ", []);
+help(restart) ->
+    io:format("
+    restart(ServerName)
+    --------------
+
+    Restart a couch_index_server with given ServerName and wait for
+    replacement process to start.
+    ---
+    ", []);
+help(Unknown) ->
+    io:format("Unknown function: `~p`. Please try one of the following:~n", [Unknown]),
+    [io:format("    - ~s~n", [Function]) || Function <- help()],
+    io:format("    ---~n", []),
+    ok.
+
+-spec names() -> [process_name()].
+
+names() ->
+    couch_index_server:names().
+
+-spec print_linked_processes() -> ok.
+
+print_linked_processes() ->
+    couch_debug:print_linked_processes(couch_index_server).
+
+-spec busy(Thershold :: pos_integer()) ->

Review comment:
       done in https://github.com/apache/couchdb/pull/3933/commits/6c7a1acd10f10bdf46489877930c89fd1f142782




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] rnewson commented on pull request #3933: Debug for sharded index server

Posted by GitBox <gi...@apache.org>.
rnewson commented on pull request #3933:
URL: https://github.com/apache/couchdb/pull/3933#issuecomment-1044454660


   not a refactor :P


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org