Performance and Statistics Module Re: CouchDB 0.9 and 1.0

[Damien Katz <da...@gmail.com>]

So we need ways to measure and monitor the performance of various  
CouchDB components.

Optimizations of core CouchDB code hasn't begun yet. I did spend a  
small time profiling CouchDB using the built-in Erlang tools, but  
using the Erlang tools collecting the data into something coherent was  
a challenge.

I think we need to add performance measuring code in our source. What  
this will require is probes in the code to generate statistics (time  
taken, values process, bytes written, errors encountered, etc),  
emitting the values to a statistics collection process. I'm thinking  
we can use DTRACE, but if not, we can build an erlang module pretty  
easily to do something similar.

Then as we optimize code, we place probes in the high levels of code,  
then measure, change, add probes, measure, change... recursing down  
through the lower levels of code.

One reason not to use DTRACE, is we want the performance stats  
available to admins in production settings. Maybe we need both low  
level DTRACE profiling, and high level performance monitoring, though  
it would be nice if we could combine them.

On Jul 2, 2008, at 3:08 AM, Jan Lehnardt wrote:

> Hello everybody,
> this thread is meant to collect missing work items (features and
> bugs) for for our 1.0 release and a discussion about how to split
> them up between 0.9 and 1.0.
>
> Take it away: Damien.
>
> Cheers
> Jan
> --

Re: Performance and Statistics Module Re: CouchDB 0.9 and 1.0

[Jan Lehnardt <ja...@apache.org>]

On Jul 3, 2008, at 21:34, Damien Katz wrote:

> Optimizations of core CouchDB code hasn't begun yet. I did spend a  
> small time profiling CouchDB using the built-in Erlang tools, but  
> using the Erlang tools collecting the data into something coherent  
> was a challenge.

I posted about the traces and coverage analysis earlier today and I  
hope that we get a few of the Erlang folks to have a look at that. Joe  
announced interest in these numbers but I don't know if he will give  
us any feedback :-)


> One reason not to use DTRACE, is we want the performance stats  
> available to admins in production settings.

As far as I understand, DTRACE is designed to be just that, a tool for  
admins (and developers of course). A reason why not to use DTrace is  
support for platforms that don't have DTrace. Namely Linux and  
Windows. So I think an internal  modules that gathers runtime  
statistics is still a good idea. So we can see high level things like,  
queries executed, caches used, data transferred and so on.

I'd be willing to help writing such a module since I've done a  
conceptually similar thing already for the runtimeconfig branch.

Cheers
Jan
--

> On Jul 2, 2008, at 3:08 AM, Jan Lehnardt wrote:
>
>> Hello everybody,
>> this thread is meant to collect missing work items (features and
>> bugs) for for our 1.0 release and a discussion about how to split
>> them up between 0.9 and 1.0.
>>
>> Take it away: Damien.
>>
>> Cheers
>> Jan
>> --
>
>