debugging

[Dave Cottlehuber <dc...@jsonified.com>]

Hi there,

I wanted to throw in a brief set of notes on debugging stuff I learned
about, some recently & some taken from the internets.

most of this is for running "couchdb -i" but you can also do this via
remsh if you set a cookie and either sname/name up. Let me know if
that could be covered too.

# tell a non-networked erlang vm (e.g. started by couchdb -i) to get connected

    net_kernel:start([<newname>, shortnames]).
    erlang:set_cookie(node(), 'yumyumyum').

now you can remsh or redbug to it. (what's redbug??) ….

# killing off a process (e.g. a view server) that you don't want
running but there's no API to allow you to do that. The erlang shotgun
is here!

    i(). %% look for the process name. You could do this nicer with
list comprehensions etc below.
    exit(list_to_pid("<0.114.0>"), false). %% boom!

# get a list of all running processes and then do evil stuff

- in this case we find out who is the memory hog, using v(-1) to refer
to the result of the last computation in the shell and keeping the
info around in Procs just in case we want to check it later.

    %% this trick via Geoff Cant / @archaelus
    Procs =  [ {Pid, element(2, process_info(Pid, memory))}
            || Pid <- erlang:processes() ].

     hd( lists:reverse( lists:keysort( 2, v(-1) ) ) ).

# trace all calls going through a particular module

NB this is pretty savage if you get it wrong, you can *easily* crash
the vm. redbug is usually a better option, but I don't see how to get
it to do this nifty trick:

   open_tracer(Module) ->
         dbg:tracer(),
         dbg:p(all, [call]),
         dbg:tpl(Module, [{'_', [], [{return_trace}]}]).

# redbug

If you're using erlang and you don't know about redbug you are doing it wrong!!

Install redbug as part of eper kit, build is straightforwards using
rebar, add it into your ebin path. Again I can provide more details on
that if needed.

https://github.com/massemanet/eper/blob/master/doc/redbug.txt

Let's do a simple example - tracing all calls to lists:max, from our shell:

(akai@akai)20> redbug:start("lists:max").
ok
(akai@akai)21> lists:max([100,900,300,100,720]).

08:57:40 <{erlang,apply,2}> {lists,max,[[100,900,300,100,720]]}
900

08:57:40 <{erlang,apply,2}> {lists,max,[[900,300,100,720],100]}

08:57:40 <{erlang,apply,2}> {lists,max,[[300,100,720],900]}

08:57:40 <{erlang,apply,2}> {lists,max,[[100,720],900]}

08:57:40 <{erlang,apply,2}> {lists,max,[[720],900]}

08:57:40 <{erlang,apply,2}> {lists,max,[[],900]}

cool! you can see your recursion in action.

Fancy,  how about doing that from a remote node?

switch to your shell & set up redbug to call from a normal prompt/terminal.

launch redbug from your terminal:

% redbug akai@akai 5000 10 "lists:max"

this tetls it to connect to erlang host akai@akai, with a timeout of 5
seconds (you'll likely need longer the first few times), and collect a
maximum of 10 debug traces before stopping. Trace only lists:max.

redbug also accepts cookies and a bunch of other self-explanatory options.

whip back quickly into your erl shell and up arrow to restart the
lists:max function.

Your terminal will show the same goodies.

Now imagine doing the same thing on a live system, tracing a specific
call through mochiweb layer, or something in couch docs storage
system. Woot!!!

redbug can also optionally provide the call stack, arguments and
results but that starts getting hefty. So then write the results out
to a file.

Why redbug?

- ease of use compared to writing tracers yourself
- excellent logging
- filtering by time and number of results tends to avoid the "oops my
real-time debugging killed your couches".

# stats inside the vm

A number of options - use a fancy tool like vmstats from mononcqc or
for a more hands-on view use entop.


# debugging erlang startup

sometime erlang barfs on startup. how to proceed?

erl -init_debug  -env ERL_LIB
/usr/local/Cellar/couchdb/1.2.0/lib/couchdb/erlang/lib -couch_ini

And then hand-start each application:

application:start(crypto).
application:start(couch).
…

Sometimes this helps find the issue. A working sample is here
https://friendpaste.com/6zett3aAHSafRNZbgfMBoD

I should try the last step broken down into subsidiary steps rather
than a single application.

Anybody got other tips/tricks to share?

A+
Dave

Fwd: debugging

[Jan Lehnardt <ja...@apache.org>]

This is useful for any CouchDB developer :)

Thanks Dave!
Jan
--


Begin forwarded message:

> From: Dave Cottlehuber <dc...@jsonified.com>
> Subject: debugging
> Date: November 5, 2012 09:11:49 GMT+01:00
> To: erlang@couchdb.apache.org
> Reply-To: erlang@couchdb.apache.org
> 
> Hi there,
> 
> I wanted to throw in a brief set of notes on debugging stuff I learned
> about, some recently & some taken from the internets.
> 
> most of this is for running "couchdb -i" but you can also do this via
> remsh if you set a cookie and either sname/name up. Let me know if
> that could be covered too.
> 
> # tell a non-networked erlang vm (e.g. started by couchdb -i) to get connected
> 
>    net_kernel:start([<newname>, shortnames]).
>    erlang:set_cookie(node(), 'yumyumyum').
> 
> now you can remsh or redbug to it. (what's redbug??) ….
> 
> # killing off a process (e.g. a view server) that you don't want
> running but there's no API to allow you to do that. The erlang shotgun
> is here!
> 
>    i(). %% look for the process name. You could do this nicer with
> list comprehensions etc below.
>    exit(list_to_pid("<0.114.0>"), false). %% boom!
> 
> # get a list of all running processes and then do evil stuff
> 
> - in this case we find out who is the memory hog, using v(-1) to refer
> to the result of the last computation in the shell and keeping the
> info around in Procs just in case we want to check it later.
> 
>    %% this trick via Geoff Cant / @archaelus
>    Procs =  [ {Pid, element(2, process_info(Pid, memory))}
>            || Pid <- erlang:processes() ].
> 
>     hd( lists:reverse( lists:keysort( 2, v(-1) ) ) ).
> 
> # trace all calls going through a particular module
> 
> NB this is pretty savage if you get it wrong, you can *easily* crash
> the vm. redbug is usually a better option, but I don't see how to get
> it to do this nifty trick:
> 
>   open_tracer(Module) ->
>         dbg:tracer(),
>         dbg:p(all, [call]),
>         dbg:tpl(Module, [{'_', [], [{return_trace}]}]).
> 
> # redbug
> 
> If you're using erlang and you don't know about redbug you are doing it wrong!!
> 
> Install redbug as part of eper kit, build is straightforwards using
> rebar, add it into your ebin path. Again I can provide more details on
> that if needed.
> 
> https://github.com/massemanet/eper/blob/master/doc/redbug.txt
> 
> Let's do a simple example - tracing all calls to lists:max, from our shell:
> 
> (akai@akai)20> redbug:start("lists:max").
> ok
> (akai@akai)21> lists:max([100,900,300,100,720]).
> 
> 08:57:40 <{erlang,apply,2}> {lists,max,[[100,900,300,100,720]]}
> 900
> 
> 08:57:40 <{erlang,apply,2}> {lists,max,[[900,300,100,720],100]}
> 
> 08:57:40 <{erlang,apply,2}> {lists,max,[[300,100,720],900]}
> 
> 08:57:40 <{erlang,apply,2}> {lists,max,[[100,720],900]}
> 
> 08:57:40 <{erlang,apply,2}> {lists,max,[[720],900]}
> 
> 08:57:40 <{erlang,apply,2}> {lists,max,[[],900]}
> 
> cool! you can see your recursion in action.
> 
> Fancy,  how about doing that from a remote node?
> 
> switch to your shell & set up redbug to call from a normal prompt/terminal.
> 
> launch redbug from your terminal:
> 
> % redbug akai@akai 5000 10 "lists:max"
> 
> this tetls it to connect to erlang host akai@akai, with a timeout of 5
> seconds (you'll likely need longer the first few times), and collect a
> maximum of 10 debug traces before stopping. Trace only lists:max.
> 
> redbug also accepts cookies and a bunch of other self-explanatory options.
> 
> whip back quickly into your erl shell and up arrow to restart the
> lists:max function.
> 
> Your terminal will show the same goodies.
> 
> Now imagine doing the same thing on a live system, tracing a specific
> call through mochiweb layer, or something in couch docs storage
> system. Woot!!!
> 
> redbug can also optionally provide the call stack, arguments and
> results but that starts getting hefty. So then write the results out
> to a file.
> 
> Why redbug?
> 
> - ease of use compared to writing tracers yourself
> - excellent logging
> - filtering by time and number of results tends to avoid the "oops my
> real-time debugging killed your couches".
> 
> # stats inside the vm
> 
> A number of options - use a fancy tool like vmstats from mononcqc or
> for a more hands-on view use entop.
> 
> 
> # debugging erlang startup
> 
> sometime erlang barfs on startup. how to proceed?
> 
> erl -init_debug  -env ERL_LIB
> /usr/local/Cellar/couchdb/1.2.0/lib/couchdb/erlang/lib -couch_ini
> 
> And then hand-start each application:
> 
> application:start(crypto).
> application:start(couch).
> …
> 
> Sometimes this helps find the issue. A working sample is here
> https://friendpaste.com/6zett3aAHSafRNZbgfMBoD
> 
> I should try the last step broken down into subsidiary steps rather
> than a single application.
> 
> Anybody got other tips/tricks to share?
> 
> A+
> Dave