simple profiling capability

[E R <pc...@gmail.com>]

Hi,

I have the need for a simple profiling capability for mod_perl applications.
At a few (< 50) points in my code I want to call something like:

$profiler->mark("some identifier");

and increment the counter for "some identifier". Later I want to be able to get
(through a web page) a summary (grouped by identifiers) of all the
profiling calls
made (of course, made by all Apache child processes.) For instance, if I want
to profile the number of times a particular subroutine was called, I would use:

sub mysubroutine {
  $profiler->mark("in mysubroutine");
  ...
}

This is very similar to logging, except for the summary part. Does anyone
have ideas of a good way to implement this? Shared memory? An external
daemon to record these calls? Has someone already implemented this?
Robust and simple are good.

Thanks,

ER

Re: simple profiling capability

[Perrin Harkins <pe...@elem.com>]

On Nov 12, 2007 10:59 AM, E R <pc...@gmail.com> wrote:
> 1. I don't want to profile every single subroutine. In fact, I want to
> profile only specific locations within my code, and a subroutine may
> have more than one profiling location.

You can control which packages are profiled.  Check the package_filter option.

> 2. Can I get real-time results?

You would need to feed the output file to the program that displays
the results.  You could rig something to do it after every request if
you really want to.

- Perrin

Re: simple profiling capability

[E R <pc...@gmail.com>]

On Nov 12, 2007 9:51 AM, Geoffrey Young <ge...@modperlcookbook.org> wrote:
> this should help:
>
> http://search.cpan.org/dist/Devel-Profiler/lib/Devel/Profiler/Apache.pm

The two problems I see with this package are:

1. I don't want to profile every single subroutine. In fact, I want to
profile only specific locations within my code, and a subroutine may
have more than one profiling location.

2. Can I get real-time results?

Re: simple profiling capability

[Geoffrey Young <ge...@modperlcookbook.org>]

E R wrote:
> Hi,
> 
> I have the need for a simple profiling capability for mod_perl applications.
> At a few (< 50) points in my code I want to call something like:
> 
> $profiler->mark("some identifier");
> 
> and increment the counter for "some identifier". Later I want to be able to get
> (through a web page) a summary (grouped by identifiers) of all the
> profiling calls
> made (of course, made by all Apache child processes.) For instance, if I want
> to profile the number of times a particular subroutine was called, I would use:
> 
> sub mysubroutine {
>   $profiler->mark("in mysubroutine");
>   ...
> }
> 
> This is very similar to logging, except for the summary part. Does anyone
> have ideas of a good way to implement this? Shared memory? An external
> daemon to record these calls? Has someone already implemented this?
> Robust and simple are good.

this should help:

http://search.cpan.org/dist/Devel-Profiler/lib/Devel/Profiler/Apache.pm

--Geoff

Re: simple profiling capability

[Perrin Harkins <pe...@elem.com>]

On Nov 12, 2007 3:25 PM, E R <pc...@gmail.com> wrote:
> 1. a call to $profile->mark(...) will emit a special record to the log file
> 2. on server start up, spawn a single process to monitor the log file
> and collect profile records
> 3. have the monitoring process listen on a socket for requests for the info
> 4. implement some method of keeping the monitoring process alive

You could just give it a web interface.  Write a program that watches
and summarizes the log file and writes out stats to a separate HTML
file every few seconds which you can display through your web server.

> I especially like the idea that logging won't lock up the httpd process.
> And Apache makes sure that log writes are atomic, right?

Yes, if you're just writing to a standard log.  If you use small
writes (less than 4K is typical I think, but it's OS-dependent),
writing to any file opened for append will be atomic.

- Perrin

Re: simple profiling capability

[E R <pc...@gmail.com>]

On Nov 12, 2007 11:25 AM, Jonathan Vanasco <jv...@2xlp.com> wrote:
> i'm just tossing this idea out...

Thanks for the ideas. The main issue I'm trying to find a good
solution for is the collection of the profiling info from all the
child processes because occasionally I want to poll the stats in real
time to see what they are.

The more I think about it, the more I think this might be a good solution:

1. a call to $profile->mark(...) will emit a special record to the log file
2. on server start up, spawn a single process to monitor the log file
and collect profile records
3. have the monitoring process listen on a socket for requests for the info
4. implement some method of keeping the monitoring process alive

I especially like the idea that logging won't lock up the httpd process.
And Apache makes sure that log writes are atomic, right?
Log files can get rolled, but a period check on the inode can detect
that, right?

Has something like this been implemented already?

Re: simple profiling capability

[Jonathan Vanasco <jv...@2xlp.com>]

i'm just tossing this idea out...

1) have a profiler package that handles all the logging, etc - and  
uses a constant

	package MyApp:::Profiler;
	use constant DO_PROFILE=> 1;
	sub profile {
		my ( $marker )= @_ ;
		#log;
	}
	1;

2) in your app code, do this:
	MyApp:::Profiler::DO_PROFILE && MyApp:::Profiler::profile( 'some  
identifier' );

My rationale:
	a) creating the namespace / etc means you can just wrap whatever  
method you want cleanly
	b) mp will optimize away the calls to profile if you start the  
server with DO_PROFILE set to 0 -- this way you don't have to deal  
with a performance hit in production and can leave the calls intact  
as-is.

in terms of actually recording...

i'd probably stack everything into a local variable as an array ;  
then id drop in a cleanup handler that calls a function to write it  
to disk or memory somehow... you'd have to have a reset within the  
package that clears out the var too

	my 	@profile;
	sub profile {
		my ( $marker )= @_ ;
		push @profile , $marker;
	}
	sub cleanup {
		# record
			record( \@profile );
		# reset
		@profile= ();
	}

in any event , the bulk of my suggestions are this:
	1- there's no reason to continually write to disk or shared mem , or  
call misc functions  ( unless you're profiling in-time ).  just store  
that stuff in a perl structure and dump 1x.
	2- use the optimize-away hack with constants to quickly turn off  
your debug code



On Nov 12, 2007, at 10:42 AM, E R wrote:

> Hi,
>
> I have the need for a simple profiling capability for mod_perl  
> applications.
> At a few (< 50) points in my code I want to call something like:
>
> $profiler->mark("some identifier");
>
> and increment the counter for "some identifier". Later I want to be  
> able to get
> (through a web page) a summary (grouped by identifiers) of all the
> profiling calls
> made (of course, made by all Apache child processes.) For instance,  
> if I want
> to profile the number of times a particular subroutine was called,  
> I would use:
>
> sub mysubroutine {
>   $profiler->mark("in mysubroutine");
>   ...
> }
>
> This is very similar to logging, except for the summary part. Does  
> anyone
> have ideas of a good way to implement this? Shared memory? An external
> daemon to record these calls? Has someone already implemented this?
> Robust and simple are good.
>
> Thanks,
>
> ER