You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avalon.apache.org by Leif Mortenson <le...@silveregg.co.jp> on 2002/03/03 16:56:16 UTC
Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Alternate Profiler Proposal.
I spent all last week working through the issues of profiling and came up with an API
which is quite a bit different that the current API in the profile package.
I have checked in a new API in the altprofile package in the scratchpad.
I had the goal of coming up with a system which would make it easy to design
Profilability into any component (Note lower case ‘c’) without having to worry
about performance. This is the same thinking that has gone into many logging systems
of late. You add lots of debug logging to classes to make it easier to debug them at
a later date. This profiling system follows the same line of thinking.
A component can be made profiler ready by implementing the Profilable interface.
This interface allows a component to expose a number of ProfilePoints as well as
tell the Profiler about any child Profilables.
ProfilePoints are extremely light weight when they are not connected to the Profiler.
So components can add ProfilePoints without having to worry about a performance hit
when the Profiler is not in use.
This implementation has been divided into 4 packages:
1) excalibur.altprofile: Defines the base interfaces and classes that most components
will directly make use of.
2) excalibur.altprofile.component: Defines a new ProfilerComponentManager which extends
the ExcaliburComponentManager and understands how to deal with Profilable components.
3) excalibur.altprofile.profiler: This package contains the guts of the system. It
defines a ProfilerManager which is where all the work of profiling an Avalon
application is handled.
4) excalibur.altprofile.profiler.gui: This package contains a reference implementation
of a simple GUI which enables you to browse and view the profiler aware elements of
an application.
The excalibur.altprofile.component package required that I make a couple changes to
the excalibur.component package to allow key classes to be extended.
So far, I have temporarily made the ResourceLimitingJdbcDataSource in the scratchpad
Profilable to make it easy to test things out. The ProfileManager also includes
default profile points for monitoring the JVM memory.
Lets start out with a sample piece of code that will start up an application using
the ProfilerComponentManager and a Profiler GUI. This code should be familiar to
anyone who has worked with the Excalibur ComponentManager:
------
DefaultContext context = new DefaultContext();
context.put("root", getServletContext().getRealPath("/"));
context.makeReadOnly();
DefaultConfigurationBuilder builder = new DefaultConfigurationBuilder();
Configuration systemConfig = builder.build(configStream);
DefaultLogKitManager logManager = new DefaultLogKitManager();
Logger lmLogger = Hierarchy.getDefaultHierarchy().getLoggerFor("log-mgr");
lmLogger.setPriority(Priority.getPriorityForName(systemConfig.getAttribute("log", "INFO")));
logManager.setLogger(lmLogger);
logManager.configure(systemConfig.getChild("logkit"));
// Set up the Profiler Manager
DefaultProfilerManager profilerManager = new DefaultProfilerManager();
profilerManager.enableLogging(new LogKitLogger(logManager.getLogger("pm")));
profilerManager.configure(systemConfig.getChild("profiler"));
profilerManager.initialize();
// Set up the Role Manager
Configuration roleConfig = builder.build(roleStream);
DefaultRoleManager roles = new DefaultRoleManager();
roles.setLogger(logManager.getLogger("rm"));
roles.configure(roleConfig);
// Set up the Component Manager
ProfilerComponentManager manager = new ProfilerComponentManager();
manager.setLogger(logManager.getLogger("cm"));
manager.setLogKitManager(logManager);
manager.contextualize(context);
manager.setProfilerManager(profilerManager);
manager.setRoleManager(roles);
manager.configure(systemConfig.getChild("environment"));
manager.initialize();
// Set up the ProfilerFrame
ProfilerFrame profilerFrame = new ProfilerFrame( profilerManager, "Nikko System Profiler" );
profilerFrame.setVisible( true );
------
The differences are that this code creates a ProfilerManager and assigns it to the
ProfilerComponentManager using the setProfilerManager() method. The code then
continues to create a ProfilerFrame passing the ProfilerManager to the constructor.
The ProfilerFrame is not required by the Profiler. It is just a default way to view
the Profile data.
Without any other changes, your application should be able to be run using the above
code to launch the application. But at this point none of the Profilable components
will be profiled because they have not been configured in the config file yet.
Components which implement Profilable are not added to the list of components to be
profiled by default. To make them profilable, you must add a ‘profilable’
attribute which gives them a profiler name.
Here is an example datasource configuration:
------
<datasources>
<jdbc name="asp-service" profilable="jdbc-asp-service">
<pool-controller min="1" max="10"/>
<auto-commit>true</auto-commit>
<driver>org.postgresql.Driver</driver>
<dburl>jdbc:postgresql://host/asp_service</dburl>
<user>user</user>
<password>pwd</password>
</jdbc>
</datasources>
------
Assuming the roles is configured to use a ResourceLimitingJdbcDataSource for the above
datasource, the profiler will now recognize the DataSource as a Profilable named
'jdbc-asp-service'.
An application can now request to be added as a listener of update events for any one
of 3 ProflilePoints published by the DataSource.
To make things work really nicely though, we also need to configure the Profilable
using the profiler configuration. It takes on the following format:
------
<profiler>
<profilables>
<profilable name="profiler"
description="Profiler">
<profile-point name="total-memory"
description="Total Memory">
<sample type="max" interval="1000" size="600"
description="Maximum each second."/>
<sample type="max" interval="60000" size="1440"
description="Maximum each minute."/>
<sample type="max" interval="3600000" size="720"
description="Maximum each hour."/>
</profile-point>
<profile-point name="free-memory"
description="Free Memory">
<sample type="min" interval="1000" size="600"
description="Minimum each second."/>
<sample type="min" interval="60000" size="1440"
description="Minimum each minute."/>
<sample type="min" interval="3600000" size="720"
description="Minimum each hour."/>
</profile-point>
<profile-point name="memory"
description="In-Use Memory">
<sample type="max" interval="1000" size="600"
description="Maximum each second."/>
<sample type="max" interval="60000" size="1440"
description="Maximum each minute."/>
<sample type="max" interval="3600000" size="720"
description="Maximum each hour."/>
</profile-point>
</profilable>
<profilable name="jdbc-asp-service"
description="ASP Service JDBC Pool">
<profile-point name="pool.new-poolables"
description="New Connections">
<sample type="ctr" interval="1000" size="600"
description="Count / second."/>
<sample type="ctr" interval="60000" size="1440"
description="Count / minute."/>
<sample type="ctr" interval="3600000" size="720"
description="Count / hour."/>
</profile-point>
<profile-point name="pool.size"
description="Open Connections">
<sample type="max" interval="1000" size="600"
description="Maximum each second."/>
<sample type="max" interval="60000" size="1440"
description="Maximum each minute."/>
<sample type="max" interval="3600000" size="720"
description="Maximum each hour."/>
</profile-point>
<profile-point name="pool.ready-size"
description="Available Connections">
<sample type="max" interval="1000" size="600"
description="Maximum available each second."/>
<sample type="max" interval="60000" size="1440"
description="Maximum available each minute."/>
<sample type="max" interval="3600000" size="720"
description="Maximum available each hour."/>
<sample type="min" interval="1000" size="600"
description="Minimum available each second."/>
<sample type="min" interval="300000" size="288"
description="Minimum available each minute."/>
<sample type="min" interval="3600000" size="720"
description="Minimum available each hour."/>
</profile-point>
</profilable>
</profilables>
</profiler>
------
This example profiler configure configures two Profilables. The first called "profiler"
configures the ProfilerManager. The second, “jdbc-asp-service” configures our example
DataSource.
By default, ProfilePoints are purely pull based profiling objects. In order to make the
profiling data available for a user to see at any time, the data must be collected and
stored into history. This is done by defining one or more Samples for each Profile Point.
The system comes with two types of ProfilePoints. The first, CounterProfilePoint, can be
used to profile the number of times an action takes place. It exposes an increment()
method. The second, ValueProfilePoint, is used for profiling absolute values. Things
like pool sizes, etc. It exposes a setValue(int value) method.
CounterProfilePoints can be assigned samples of type “counter” (“ctr”).
ValueProfilePoints can be assigned samples of types “maximum” (“max”),
“minimum” (“min”), and “average” (“avg”).
Taking the “maximum” types sample as an example. It will monitor each call to setValue
for the ProfilePoint and store the maximum value in a particular sample period as the
value. “minimum” stores the minimum value, and “average”, the average value for the
sample period. The “counter” sample will store the number of times that increment()
was called on the ProfilePoint during the sample period.
The ProfilePoints which have samples visible will be viewable as graphs in real time
using the ProfilerFrame class.
Right now, this shouldn’t be too difficult to get set up. But I get a example
application that will work out of the box very soon.
Wanted to get something up as soon as possible though because it looks like Marcus
has also been giving this a lot of thought.
Cheers, I am sure you will have some questions about how things work. It is late
here now. I should be able to answer them in the AM.
Cheers,
Leif
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Berin Loritsch <bl...@apache.org>.
Leif Mortenson wrote:
> Alternate Profiler Proposal.
>
> I spent all last week working through the issues of profiling and came up with an API
> which is quite a bit different that the current API in the profile package.
>
> I have checked in a new API in the altprofile package in the scratchpad.
Excellent!
> I had the goal of coming up with a system which would make it easy to design
> Profilability into any component (Note lower case ‘c’) without having to worry
> about performance. This is the same thinking that has gone into many logging systems
> of late. You add lots of debug logging to classes to make it easier to debug them at
> a later date. This profiling system follows the same line of thinking.
Ok. My approach was to send a startProfiling() message to the
Profilables being monitored, and then send an endProfiling() message
when it is all done.
However, I am excited to have a working tool available.
> A component can be made profiler ready by implementing the Profilable interface.
> This interface allows a component to expose a number of ProfilePoints as well as
> tell the Profiler about any child Profilables.
So far we are in sync.
> ProfilePoints are extremely light weight when they are not connected to the Profiler.
> So components can add ProfilePoints without having to worry about a performance hit
> when the Profiler is not in use.
Yep. Mine too. The biggest difference I believe is the relationship of
ProfileReport and Profiler.
> This implementation has been divided into 4 packages:
>
> 1) excalibur.altprofile: Defines the base interfaces and classes that most components
> will directly make use of.
Excellent.
> 2) excalibur.altprofile.component: Defines a new ProfilerComponentManager which extends
> the ExcaliburComponentManager and understands how to deal with Profilable components.
Kool. I will most likely throw in something like this in the Container/
ContainerManager stuff as well. I will be refactoring a few things
to make that code easier to use though...
> 3) excalibur.altprofile.profiler: This package contains the guts of the system. It
> defines a ProfilerManager which is where all the work of profiling an Avalon
> application is handled.
Excellent.
> 4) excalibur.altprofile.profiler.gui: This package contains a reference implementation
> of a simple GUI which enables you to browse and view the profiler aware elements of
> an application.
"We're not worthy! We're not worthy!"
Great work guys. I mean it.
> The excalibur.altprofile.component package required that I make a couple changes to
> the excalibur.component package to allow key classes to be extended.
That's fine.
> So far, I have temporarily made the ResourceLimitingJdbcDataSource in the scratchpad
> Profilable to make it easy to test things out. The ProfileManager also includes
> default profile points for monitoring the JVM memory.
Excellent.
> Lets start out with a sample piece of code that will start up an application using
> the ProfilerComponentManager and a Profiler GUI. This code should be familiar to
> anyone who has worked with the Excalibur ComponentManager:
>
> ------
> DefaultContext context = new DefaultContext();
>
> context.put("root", getServletContext().getRealPath("/"));
> context.makeReadOnly();
>
> DefaultConfigurationBuilder builder = new DefaultConfigurationBuilder();
> Configuration systemConfig = builder.build(configStream);
>
> DefaultLogKitManager logManager = new DefaultLogKitManager();
> Logger lmLogger = Hierarchy.getDefaultHierarchy().getLoggerFor("log-mgr");
> lmLogger.setPriority(Priority.getPriorityForName(systemConfig.getAttribute("log", "INFO")));
> logManager.setLogger(lmLogger);
> logManager.configure(systemConfig.getChild("logkit"));
>
>
> // Set up the Profiler Manager
> DefaultProfilerManager profilerManager = new DefaultProfilerManager();
>
> profilerManager.enableLogging(new LogKitLogger(logManager.getLogger("pm")));
> profilerManager.configure(systemConfig.getChild("profiler"));
> profilerManager.initialize();
>
> // Set up the Role Manager
> Configuration roleConfig = builder.build(roleStream);
> DefaultRoleManager roles = new DefaultRoleManager();
>
> roles.setLogger(logManager.getLogger("rm"));
> roles.configure(roleConfig);
>
>
> // Set up the Component Manager
> ProfilerComponentManager manager = new ProfilerComponentManager();
>
> manager.setLogger(logManager.getLogger("cm"));
> manager.setLogKitManager(logManager);
> manager.contextualize(context);
> manager.setProfilerManager(profilerManager);
> manager.setRoleManager(roles);
> manager.configure(systemConfig.getChild("environment"));
> manager.initialize();
>
> // Set up the ProfilerFrame
> ProfilerFrame profilerFrame = new ProfilerFrame( profilerManager, "Nikko System Profiler" );
> profilerFrame.setVisible( true );
> ------
>
> The differences are that this code creates a ProfilerManager and assigns it to the
> ProfilerComponentManager using the setProfilerManager() method. The code then
> continues to create a ProfilerFrame passing the ProfilerManager to the constructor.
> The ProfilerFrame is not required by the Profiler. It is just a default way to view
> the Profile data.
By the way, It was suggested to me a while back that we should rename
the Profiler stuff as Instrumentor, InstrumentationPoint, etc. The
reason being that we are not truly profiling, but instrumenting the
system. I would like to get this code up and running and final as
soon as possible. (I have a book to write, and this can really make
an impact).
> Without any other changes, your application should be able to be run using the above
> code to launch the application. But at this point none of the Profilable components
> will be profiled because they have not been configured in the config file yet.
>
> Components which implement Profilable are not added to the list of components to be
> profiled by default. To make them profilable, you must add a ‘profilable’
> attribute which gives them a profiler name.
>
> Here is an example datasource configuration:
> ------
> <datasources>
> <jdbc name="asp-service" profilable="jdbc-asp-service">
> <pool-controller min="1" max="10"/>
> <auto-commit>true</auto-commit>
> <driver>org.postgresql.Driver</driver>
> <dburl>jdbc:postgresql://host/asp_service</dburl>
> <user>user</user>
> <password>pwd</password>
> </jdbc>
> </datasources>
> ------
Ok. Looks good. We *may* want to make that alterable during run-time
though. For instance, the GUI may want to select specific Profilables
that we are monitoring at any one time.
> Assuming the roles is configured to use a ResourceLimitingJdbcDataSource for the above
> datasource, the profiler will now recognize the DataSource as a Profilable named
> 'jdbc-asp-service'.
>
> An application can now request to be added as a listener of update events for any one
> of 3 ProflilePoints published by the DataSource.
>
> To make things work really nicely though, we also need to configure the Profilable
> using the profiler configuration. It takes on the following format:
>
> ------
> <profiler>
> <profilables>
> <profilable name="profiler"
> description="Profiler">
> <profile-point name="total-memory"
> description="Total Memory">
> <sample type="max" interval="1000" size="600"
> description="Maximum each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum each hour."/>
> </profile-point>
>
> <profile-point name="free-memory"
> description="Free Memory">
> <sample type="min" interval="1000" size="600"
> description="Minimum each second."/>
> <sample type="min" interval="60000" size="1440"
> description="Minimum each minute."/>
> <sample type="min" interval="3600000" size="720"
> description="Minimum each hour."/>
> </profile-point>
>
> <profile-point name="memory"
> description="In-Use Memory">
> <sample type="max" interval="1000" size="600"
> description="Maximum each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum each hour."/>
> </profile-point>
> </profilable>
>
> <profilable name="jdbc-asp-service"
> description="ASP Service JDBC Pool">
> <profile-point name="pool.new-poolables"
> description="New Connections">
> <sample type="ctr" interval="1000" size="600"
> description="Count / second."/>
> <sample type="ctr" interval="60000" size="1440"
> description="Count / minute."/>
> <sample type="ctr" interval="3600000" size="720"
> description="Count / hour."/>
> </profile-point>
>
> <profile-point name="pool.size"
> description="Open Connections">
> <sample type="max" interval="1000" size="600"
> description="Maximum each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum each hour."/>
> </profile-point>
>
> <profile-point name="pool.ready-size"
> description="Available Connections">
> <sample type="max" interval="1000" size="600"
> description="Maximum available each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum available each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum available each hour."/>
>
> <sample type="min" interval="1000" size="600"
> description="Minimum available each second."/>
> <sample type="min" interval="300000" size="288"
> description="Minimum available each minute."/>
> <sample type="min" interval="3600000" size="720"
> description="Minimum available each hour."/>
> </profile-point>
> </profilable>
> </profilables>
> </profiler>
> ------
I'm kind of luke-warm on this. We should be able to determine the type
of sample we are accepting automatically. The reporting agent (whether
gui or file) should be able to adjust the report to reflect the per
second, per minute, per hour differences. Especially since it is
controlling the sampling frequency directly.
> This example profiler configure configures two Profilables. The first called "profiler"
> configures the ProfilerManager. The second, “jdbc-asp-service” configures our example
> DataSource.
>
> By default, ProfilePoints are purely pull based profiling objects. In order to make the
> profiling data available for a user to see at any time, the data must be collected and
> stored into history. This is done by defining one or more Samples for each Profile Point.
>
> The system comes with two types of ProfilePoints. The first, CounterProfilePoint, can be
> used to profile the number of times an action takes place. It exposes an increment()
> method. The second, ValueProfilePoint, is used for profiling absolute values. Things
> like pool sizes, etc. It exposes a setValue(int value) method.
>
> CounterProfilePoints can be assigned samples of type “counter” (“ctr”).
> ValueProfilePoints can be assigned samples of types “maximum” (“max”),
> “minimum” (“min”), and “average” (“avg”).
>
> Taking the “maximum” types sample as an example. It will monitor each call to setValue
> for the ProfilePoint and store the maximum value in a particular sample period as the
> value. “minimum” stores the minimum value, and “average”, the average value for the
> sample period. The “counter” sample will store the number of times that increment()
> was called on the ProfilePoint during the sample period.
>
> The ProfilePoints which have samples visible will be viewable as graphs in real time
> using the ProfilerFrame class.
>
> Right now, this shouldn’t be too difficult to get set up. But I get a example
> application that will work out of the box very soon.
>
> Wanted to get something up as soon as possible though because it looks like Marcus
> has also been giving this a lot of thought.
>
> Cheers, I am sure you will have some questions about how things work. It is late
> here now. I should be able to answer them in the AM.
So far, kudos for the excellent work. I will review the interfaces
shortly, and let you know if I have any questions.
--
"They that give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety."
- Benjamin Franklin
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Raw Instrumentation Data Format [was: Commit: New AltProfile package...]
Posted by Ryan Shaw <ry...@silveregg.co.jp>.
Hello,
See comments below.
Leif wrote:
||| >I was thinking of storing the values as a large element like
||| >this (FORMAT 1):
||| >
||| ><history>10,23,23, ... ,15, 16</history>
||| >
||| >rather than this (FORMAT 2):
|||
||| ><history>
||| ><sample>10</sample>
||| ><sample>23</sample>
||| ><sample>23</sample>
||| >...
||| ><sample>15</sample>
||| ><sample>16</sample>
||| ></history>
|||
||| Ok, I went ahead and implemented this (FORMAT 1) as the default
||| behavior for the profiler manager.
||| It will read state files in either format, but the method can be set for
||| writes using the
||| use-compact-samples attribute in profiler.xml.
I think it would be better to have a standard format for the
data file, so that anyone can write an alternative tool for
processing the raw persistent instrumentation data.
The format can be versioned, to allow for changes in the future.
So these raw data files look the same (outside of versioning
differences) for all instrumented Avalon components, everywhere.
Then there can be tools for doing whatever people want with
the raw data. The normal ones can be included with Avalon
and if someone has some weird need they can write their own,
comfortable in the knowledge that it will work with other
Avalon systems, even ones running since before they wrote it.
Ryan
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Leif Mortenson <le...@silveregg.co.jp>.
Leif Mortenson wrote:
>Also, currently the state files are quite large because each of the
>sample data points are saved
>as an element. This also makes for slow loading and saving.
>
>I was thinking of storing the values as a large element like this:
>
><history>10,23,23, ... ,15, 16</history>
>
>rather than this:
><history>
><sample>10</sample>
><sample>23</sample>
><sample>23</sample>
>...
><sample>15</sample>
><sample>16</sample>
></history>
>
Ok, I went ahead and implemented this as the default behavior for the
profiler manager.
It will read state files in either format, but the method can be set for
writes using the
use-compact-samples attribute in profiler.xml:
---
|<profiler logger="pm">
| <state-file interval="60000"
use-compact-samples="true">../data/profiler.sampledata</state-file>
|
| <profilables>
| ...
---
Using the compact format makes the files about 10% the size, and the
state saves and loads are
about 85% faster. For the cases where you really need the full XML data,
the attribute can be
set to false.
Leif
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Leif Mortenson <le...@silveregg.co.jp>.
Berin Loritsch wrote:
>>I was thinking of storing the values as a large element like this:
>>
>><history>10,23,23, ... ,15, 16</history>
>>
>>rather than this:
>><history>
>><sample>10</sample>
>><sample>23</sample>
>><sample>23</sample>
>>...
>><sample>15</sample>
>><sample>16</sample>
>></history>
>>
>>What are peoples thoughts on this. It is kind of non-XMLish, but would
>>it be worth it for
>>the performance and space savings you would get?
>>
>
>
> My original thoughts were that we might want to plug in different report
> types. For example: Comma Separated Value (CSV) format (I have a
> plug-in for some other software that can format strings correctly),
> Delimited Format (pipe delimited, tab delimited...), etc.
>
> Sample data should be separate and distinct from the XML. That way we
> can load it in other programs (such as M$ Excel) as we wish. The XML
> config file should reference the sample data so it can all be loaded at
> once.
There are actually two different files that I am dealing with. The first is the
Profiler configuration file. (profiler.xml) The profiler can be configured in
this file to load past sample data from a state file (profiler.sampledata) and
then make snapshots of the data back into the file at regular intervals and when
the ProfilerManager is disposed. This makes it possible to ovserve profiled
data across JVM invocations.
The file is state file is configured in the profiler.xml file like this:
----
|<profiler logger="pm">
|
<state-file interval="60000">../data/profiler.sampledata</state-file>
|
|
<profilables>
| ...
----
One thing to keep strait is that this state file does not necessarily need to
be thought of as a report. This is something that needs to be written very
efficiently at regular intervals. This way if the server goes down, a recent
state file will be there to help give some clues as to the problem.
What are your ideas for the kinds of things that you want to do with reports
etc. We should look at whether they should be done directly by the profiler
or whether there should be a report manager or something which makes use
of data collected by the profiler manager to generate the reports. I am not
yet clear on what your vision of the profiling system is and what you would
like to get out of it.
My goals were two fold. something that would be useful to debug performance
and resource issues during the development cycle. But also a tool which would
be useful to the system administrator in charge of an online system.
I can envision things like a monitor component which added itself as a listener
to one or more profile points via the profiler manager and then did things like
raise alarms or send alert mails when certain values were observed.
You could also create ProfilePoints with ProfileSamples to track the number of
times users took a certain action. The charts then make it very easy to see
trends. If it was used on an web site for example, you could have profile
points which tracked the number of times certain pages were viewed, purchaces
were made etc. The data could then be viewed at various intervals. In a way
the profiler is quite similar to a logging system.
I am rambling a bit, but its late. :-)
Cheers,
Leif
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Berin Loritsch <bl...@apache.org>.
Leif Mortenson wrote:
>
> Leif Mortenson wrote:
>
>
>>Leif Mortenson wrote:
>>
>>
>>>Alternate Profiler Proposal.
>>>
>>>
>>The altprofile project now supports saving and loading of desktop state
>>in the ProfilerFrame and
>>the automatic saving and loading of Profile Sample data state in the
>>ProfilerManager.
>>
>>Both of these features are demonstrated in the altprofile example
>>application.
>>
>>Both features are currently riding on top of the ability to save and
>>load Configuration objects
>>as XML files. I was a little concerned that this might cause confusion.
>>Any thoughts?
Configuration is perfect to be saved in XML, I have no issues with that.
> Also, currently the state files are quite large because each of the
> sample data points are saved
> as an element. This also makes for slow loading and saving.
>
> I was thinking of storing the values as a large element like this:
>
> <history>10,23,23, ... ,15, 16</history>
>
> rather than this:
> <history>
> <sample>10</sample>
> <sample>23</sample>
> <sample>23</sample>
> ...
> <sample>15</sample>
> <sample>16</sample>
> </history>
>
> What are peoples thoughts on this. It is kind of non-XMLish, but would
> it be worth it for
> the performance and space savings you would get?
My original thoughts were that we might want to plug in different report
types. For example: Comma Separated Value (CSV) format (I have a
plug-in for some other software that can format strings correctly),
Delimited Format (pipe delimited, tab delimited...), etc.
Sample data should be separate and distinct from the XML. That way we
can load it in other programs (such as M$ Excel) as we wish. The XML
config file should reference the sample data so it can all be loaded at
once.
--
"They that give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety."
- Benjamin Franklin
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Leif Mortenson <le...@silveregg.co.jp>.
Leif Mortenson wrote:
>Leif Mortenson wrote:
>
>>Alternate Profiler Proposal.
>>
>The altprofile project now supports saving and loading of desktop state
>in the ProfilerFrame and
>the automatic saving and loading of Profile Sample data state in the
>ProfilerManager.
>
>Both of these features are demonstrated in the altprofile example
>application.
>
>Both features are currently riding on top of the ability to save and
>load Configuration objects
>as XML files. I was a little concerned that this might cause confusion.
>Any thoughts?
>
Also, currently the state files are quite large because each of the
sample data points are saved
as an element. This also makes for slow loading and saving.
I was thinking of storing the values as a large element like this:
<history>10,23,23, ... ,15, 16</history>
rather than this:
<history>
<sample>10</sample>
<sample>23</sample>
<sample>23</sample>
...
<sample>15</sample>
<sample>16</sample>
</history>
What are peoples thoughts on this. It is kind of non-XMLish, but would
it be worth it for
the performance and space savings you would get?
Leif
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Leif Mortenson <le...@silveregg.co.jp>.
Leif Mortenson wrote:
>Alternate Profiler Proposal.
>
The altprofile project now supports saving and loading of desktop state
in the ProfilerFrame and
the automatic saving and loading of Profile Sample data state in the
ProfilerManager.
Both of these features are demonstrated in the altprofile example
application.
Both features are currently riding on top of the ability to save and
load Configuration objects
as XML files. I was a little concerned that this might cause confusion.
Any thoughts?
Cheers,
Leif
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an alternate Profiling system.
Posted by Marcus Crafter <cr...@fztig938.bank.dresdner.net>.
Hi Leif!
Great work. You've been busy :)
I'm currently getting up to speed with your code, I'll post some
comments soon.
Cheers,
Marcus
On Mon, Mar 04, 2002 at 12:56:16AM +0900, Leif Mortenson wrote:
> Alternate Profiler Proposal.
>
> I spent all last week working through the issues of profiling and came up with an API
> which is quite a bit different that the current API in the profile package.
>
> I have checked in a new API in the altprofile package in the scratchpad.
>
> I had the goal of coming up with a system which would make it easy to design
> Profilability into any component (Note lower case ?$B!Fc?$B!G) without having to worry
> about performance. This is the same thinking that has gone into many logging systems
> of late. You add lots of debug logging to classes to make it easier to debug them at
> a later date. This profiling system follows the same line of thinking.
>
> A component can be made profiler ready by implementing the Profilable interface.
> This interface allows a component to expose a number of ProfilePoints as well as
> tell the Profiler about any child Profilables.
>
> ProfilePoints are extremely light weight when they are not connected to the Profiler.
> So components can add ProfilePoints without having to worry about a performance hit
> when the Profiler is not in use.
>
> This implementation has been divided into 4 packages:
>
> 1) excalibur.altprofile: Defines the base interfaces and classes that most components
> will directly make use of.
>
> 2) excalibur.altprofile.component: Defines a new ProfilerComponentManager which extends
> the ExcaliburComponentManager and understands how to deal with Profilable components.
>
> 3) excalibur.altprofile.profiler: This package contains the guts of the system. It
> defines a ProfilerManager which is where all the work of profiling an Avalon
> application is handled.
>
> 4) excalibur.altprofile.profiler.gui: This package contains a reference implementation
> of a simple GUI which enables you to browse and view the profiler aware elements of
> an application.
>
>
> The excalibur.altprofile.component package required that I make a couple changes to
> the excalibur.component package to allow key classes to be extended.
>
> So far, I have temporarily made the ResourceLimitingJdbcDataSource in the scratchpad
> Profilable to make it easy to test things out. The ProfileManager also includes
> default profile points for monitoring the JVM memory.
>
> Lets start out with a sample piece of code that will start up an application using
> the ProfilerComponentManager and a Profiler GUI. This code should be familiar to
> anyone who has worked with the Excalibur ComponentManager:
>
> ------
> DefaultContext context = new DefaultContext();
>
> context.put("root", getServletContext().getRealPath("/"));
> context.makeReadOnly();
>
> DefaultConfigurationBuilder builder = new DefaultConfigurationBuilder();
> Configuration systemConfig = builder.build(configStream);
>
> DefaultLogKitManager logManager = new DefaultLogKitManager();
> Logger lmLogger = Hierarchy.getDefaultHierarchy().getLoggerFor("log-mgr");
> lmLogger.setPriority(Priority.getPriorityForName(systemConfig.getAttribute("log", "INFO")));
> logManager.setLogger(lmLogger);
> logManager.configure(systemConfig.getChild("logkit"));
>
>
> // Set up the Profiler Manager
> DefaultProfilerManager profilerManager = new DefaultProfilerManager();
>
> profilerManager.enableLogging(new LogKitLogger(logManager.getLogger("pm")));
> profilerManager.configure(systemConfig.getChild("profiler"));
> profilerManager.initialize();
>
> // Set up the Role Manager
> Configuration roleConfig = builder.build(roleStream);
> DefaultRoleManager roles = new DefaultRoleManager();
>
> roles.setLogger(logManager.getLogger("rm"));
> roles.configure(roleConfig);
>
>
> // Set up the Component Manager
> ProfilerComponentManager manager = new ProfilerComponentManager();
>
> manager.setLogger(logManager.getLogger("cm"));
> manager.setLogKitManager(logManager);
> manager.contextualize(context);
> manager.setProfilerManager(profilerManager);
> manager.setRoleManager(roles);
> manager.configure(systemConfig.getChild("environment"));
> manager.initialize();
>
> // Set up the ProfilerFrame
> ProfilerFrame profilerFrame = new ProfilerFrame( profilerManager, "Nikko System Profiler" );
> profilerFrame.setVisible( true );
> ------
>
> The differences are that this code creates a ProfilerManager and assigns it to the
> ProfilerComponentManager using the setProfilerManager() method. The code then
> continues to create a ProfilerFrame passing the ProfilerManager to the constructor.
> The ProfilerFrame is not required by the Profiler. It is just a default way to view
> the Profile data.
>
> Without any other changes, your application should be able to be run using the above
> code to launch the application. But at this point none of the Profilable components
> will be profiled because they have not been configured in the config file yet.
>
> Components which implement Profilable are not added to the list of components to be
> profiled by default. To make them profilable, you must add a ?$B!Fprofilable?$B!G
> attribute which gives them a profiler name.
>
> Here is an example datasource configuration:
> ------
> <datasources>
> <jdbc name="asp-service" profilable="jdbc-asp-service">
> <pool-controller min="1" max="10"/>
> <auto-commit>true</auto-commit>
> <driver>org.postgresql.Driver</driver>
> <dburl>jdbc:postgresql://host/asp_service</dburl>
> <user>user</user>
> <password>pwd</password>
> </jdbc>
> </datasources>
> ------
>
> Assuming the roles is configured to use a ResourceLimitingJdbcDataSource for the above
> datasource, the profiler will now recognize the DataSource as a Profilable named
> 'jdbc-asp-service'.
>
> An application can now request to be added as a listener of update events for any one
> of 3 ProflilePoints published by the DataSource.
>
> To make things work really nicely though, we also need to configure the Profilable
> using the profiler configuration. It takes on the following format:
>
> ------
> <profiler>
> <profilables>
> <profilable name="profiler"
> description="Profiler">
> <profile-point name="total-memory"
> description="Total Memory">
> <sample type="max" interval="1000" size="600"
> description="Maximum each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum each hour."/>
> </profile-point>
>
> <profile-point name="free-memory"
> description="Free Memory">
> <sample type="min" interval="1000" size="600"
> description="Minimum each second."/>
> <sample type="min" interval="60000" size="1440"
> description="Minimum each minute."/>
> <sample type="min" interval="3600000" size="720"
> description="Minimum each hour."/>
> </profile-point>
>
> <profile-point name="memory"
> description="In-Use Memory">
> <sample type="max" interval="1000" size="600"
> description="Maximum each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum each hour."/>
> </profile-point>
> </profilable>
>
> <profilable name="jdbc-asp-service"
> description="ASP Service JDBC Pool">
> <profile-point name="pool.new-poolables"
> description="New Connections">
> <sample type="ctr" interval="1000" size="600"
> description="Count / second."/>
> <sample type="ctr" interval="60000" size="1440"
> description="Count / minute."/>
> <sample type="ctr" interval="3600000" size="720"
> description="Count / hour."/>
> </profile-point>
>
> <profile-point name="pool.size"
> description="Open Connections">
> <sample type="max" interval="1000" size="600"
> description="Maximum each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum each hour."/>
> </profile-point>
>
> <profile-point name="pool.ready-size"
> description="Available Connections">
> <sample type="max" interval="1000" size="600"
> description="Maximum available each second."/>
> <sample type="max" interval="60000" size="1440"
> description="Maximum available each minute."/>
> <sample type="max" interval="3600000" size="720"
> description="Maximum available each hour."/>
>
> <sample type="min" interval="1000" size="600"
> description="Minimum available each second."/>
> <sample type="min" interval="300000" size="288"
> description="Minimum available each minute."/>
> <sample type="min" interval="3600000" size="720"
> description="Minimum available each hour."/>
> </profile-point>
> </profilable>
> </profilables>
> </profiler>
> ------
>
> This example profiler configure configures two Profilables. The first called "profiler"
> configures the ProfilerManager. The second, ?$B!Hjdbc-asp-service?$B!I configures our example
> DataSource.
>
> By default, ProfilePoints are purely pull based profiling objects. In order to make the
> profiling data available for a user to see at any time, the data must be collected and
> stored into history. This is done by defining one or more Samples for each Profile Point.
>
> The system comes with two types of ProfilePoints. The first, CounterProfilePoint, can be
> used to profile the number of times an action takes place. It exposes an increment()
> method. The second, ValueProfilePoint, is used for profiling absolute values. Things
> like pool sizes, etc. It exposes a setValue(int value) method.
>
> CounterProfilePoints can be assigned samples of type ?$B!Hcounter?$B!I (?$B!Hctr?$B!I).
> ValueProfilePoints can be assigned samples of types ?$B!Hmaximum?$B!I (?$B!Hmax?$B!I),
> ?$B!Hminimum?$B!I (?$B!Hmin?$B!I), and ?$B!Haverage?$B!I (?$B!Havg?$B!I).
>
> Taking the ?$B!Hmaximum?$B!I types sample as an example. It will monitor each call to setValue
> for the ProfilePoint and store the maximum value in a particular sample period as the
> value. ?$B!Hminimum?$B!I stores the minimum value, and ?$B!Haverage?$B!I, the average value for the
> sample period. The ?$B!Hcounter?$B!I sample will store the number of times that increment()
> was called on the ProfilePoint during the sample period.
>
> The ProfilePoints which have samples visible will be viewable as graphs in real time
> using the ProfilerFrame class.
>
> Right now, this shouldn?$B!Gt be too difficult to get set up. But I get a example
> application that will work out of the box very soon.
>
> Wanted to get something up as soon as possible though because it looks like Marcus
> has also been giving this a lot of thought.
>
> Cheers, I am sure you will have some questions about how things work. It is late
> here now. I should be able to answer them in the AM.
>
> Cheers,
> Leif
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
.....
,,$$$$$$$$$, Marcus Crafter
;$' '$$$$: Computer Systems Engineer
$: $$$$: ManageSoft GmbH
$ o_)$$$: 82-84 Mainzer Landstrasse
;$, _/\ &&:' 60327 Frankfurt Germany
' /( &&&
\_&&&&'
&&&&.
&&&&&&&:
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an alternate Profiling system.
Posted by Peter Donald <pe...@apache.org>.
On Mon, 4 Mar 2002 03:01, Leif Mortenson wrote:
> Leif Mortenson wrote:
> > Alternate Profiler Proposal.
>
> Here is a sample screen shot of what the ProfilerFrame will look like when
> it is up and running.
Wow!
That looks like fantastic work. To be honest I haven't paid much attention to
Profiling stuff in the past. Is the intention of making Cocoon and its
services profilable? That would make it much easier to profile different bits
of that system.
--
Cheers,
Pete
Doubt is not a pleasant condition, but certainty is absurd.
-- Voltaire
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Commit: New AltProfile package as reference implementation for an
alternate Profiling system.
Posted by Leif Mortenson <le...@silveregg.co.jp>.
Leif Mortenson wrote:
> Alternate Profiler Proposal.
Here is a sample screen shot of what the ProfilerFrame will look like when it is up and running.
Cheers,
Leif