Re: instrumentation and other questions

[Leif Mortenson <le...@tanukisoftware.com>]

Stephen,
At your request, moving this discussion to the dev list.

As for JMX, one thing that could be done in certain situations is to 
create an MBean for
the instrument manager which would then act as a JMX Connector to the
InstrumentManager.

Currently AltRMI and HTTP Connectors exist.   The AltRMI Connector 
exists which
allows you to connect to the IM using the Swing InstrumentClient.

The HTTP connector is not quite as convenient for letting you see more 
than one set
of sample history at a time (that can be changed).  But it does have a 
lot of benefits
as well.  Including the fact that you do not need a special client to 
connect.  Any
browser will do.  The connector can output either HTML or XML content so it
should be very easy to send the XML through Cocoon and transform the XML 
into
whatever other format you need.

Cheers,
Leif

Stephen McConnell wrote:

> Can you do me a favor and post the following extract of your email to 
> dev@avalon?
>
> My feeling is that there is a place where your work in instrumentation 
> fits with aspects of JMX and the overall merlin architecture.  There 
> are a bunch of thoughts I have about this but they are better on-list 
> than off.
>
>
> Leif Mortenson wrote:
>
>> Basically I view the Instrument package as being completely different 
>> from JMX.
>> As I understand it, using JVM you would be able to make individual 
>> data points
>> available at any given time.  But as I understand it, there is no way 
>> to build up
>> history or display the data points as they change over time.
>>
>> When I created the Instrument code.  I wanted to create a tool that 
>> would integrated
>> in a similar way as Logging is done.  Instruments would be added 
>> throughout classes
>> just as debug log output is added.  When not in use these instruments 
>> do not affect the
>> performance of the application.  But at any time, a client can 
>> connect to the JVM and
>> request a snapshot of the instrument, or ask that the instrument 
>> start building up a
>> history.
>>
>> These sample sets are requested using a lease system.   Once the 
>> sample is requested
>> they will continue to be collected in the JVM even if the JVM is 
>> restarted and or the
>> client disconnects.   It is then possible to connect at a later time, 
>> before the lease
>> expires, and see the instrument sample history that was built up.   
>> Both the HTTP
>> and instrument-client interfaces make it possible to view the sample 
>> history as a
>> graph.  By comparing key instrument sample sets.  It can be very easy 
>> to tell exactly
>> what is happening within an application.
>>
>> I make use of instruments for everything from pool sizes, to keeping 
>> track of the
>> number of times that a certain servlet is called and then how long 
>> each call takes
>> to complete.
>>
>> The instrument manager allows you to specify that certain instruments 
>> should have
>> permanent sample sets.  These do not require a lease and are always 
>> available.
>> I usually use these to track memory usage, HTTP connection and access 
>> counts
>> and other data which I want to see over a long period of time.
>>
>> Because the instrument data is preserved across JVM restarts, this 
>> makes it
>> possible to see how the memory usage of an application changes over 
>> several
>> months.
>>
>> I really need to get it documented so it would be easy for you try 
>> out.  I can
>> almost guarantee that once you understand the data that is collected 
>> you very
>> quickly find it as invaluable as logging.  My customers all use the 
>> HTTP client
>> to connect to their applications and very quickly get a feel for the 
>> health of the
>> app.   Memory leaks and resource problems are immediately obvious 
>> just by
>> glancing at the appropriate instrument sample graph.
>
>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org

Re: instrumentation and other questions

[Leif Mortenson <le...@tanukisoftware.com>]

Timothy Bennett wrote:

> Leif Mortenson wrote:
>
>> If integrated at this level, the user of the container can get a huge 
>> benefit from
>> Instrumentation without ever making use of the API themselves.
>
> I, for one, would be ecstatic to see the container internals 
> instrumented.  I agree, Leif, this would provide a great benefit. 
> Thinking out loud here, but I wonder if the container utilizes the 
> instrumentation package internally, could it be *handed* off to the 
> component via the Context so that (to the user) it would appear like 
> seamless instrumentation of both the container and component?

Both Fortress and ECM will automatically register any Components which 
implement the
Instrumentable interface.   Unlike LogEnabled, it is setup so that a 
container does not need
to support the interface.

When writing a component that instrumentable, you make your code use the 
instruments
whether the container supports them or not.   The code is optimized so 
that this does
not cause a performance hit.   Even when a component is registered with an
InstrumentManager, there is only a performance hit if and only if a 
particular instrument
is actually being monitored.  Even then, the hit is very small, although 
the later does involve
a single synchronization block.

In addition to the Instrumentable interface, a second 
InstrumentManageable interface
exists for components which need to obtain their own reference to the
InstrumenManager.   I use this for my Jetty component to provide 
instrumentation
functionality to servlets etc.   It is also useful for nested containers 
etc.

So if I understood your comment, there should be no need to make use of 
the Context.
As it is, everything already appears to be quite seamless.

Cheers,
Leif


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org

Re: instrumentation and other questions

[Timothy Bennett <ex...@comcast.net>]

Leif Mortenson wrote:
> If integrated at this level, the user of the container can get a huge 
> benefit from
> Instrumentation without ever making use of the API themselves.
> 

I, for one, would be ecstatic to see the container internals 
instrumented.  I agree, Leif, this would provide a great benefit. 
Thinking out loud here, but I wonder if the container utilizes the 
instrumentation package internally, could it be *handed* off to the 
component via the Context so that (to the user) it would appear like 
seamless instrumentation of both the container and component?


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org

Re: instrumentation and other questions

[Leif Mortenson <le...@tanukisoftware.com>]

Stephen McConnell wrote:

> Leif Mortenson wrote:
>
>> As for JMX, one thing that could be done in certain situations is to 
>> create an MBean for the instrument manager which would then act as a 
>> JMX Connector to the InstrumentManager.
>
> Thinking along similar lines.  In my head is the instrumentation API 
> as a potential reference point concerning the instrumentation 
> abstraction relative to the composition and activation frameworks.  
> This in turn could be augmented by leveraging instrumented thread and 
> pool facilities in excalibur.  Using facilities, we could add in the 
> instrumentation event transport solution - e.g. a JMX RMI adapter, 
> IIOP, AltRMI or the existing HTTP connection facilities you have in 
> place.
>
> There is one subject that I want to address:
>
>   * instrumentation of the composition and activation packages

What do you have in mind here?

Fortress does not make all that much direct use of the IM.  But I had 
been much more
involved with the ECM development and the InstrumentManager is fully 
integrated.
If you have tried it with Cocoon, you will see how useful that can be.   
You can see
information about how many times components are looked up and released 
over time.
How pools grow and shrink, as well as any blocking that may or may not 
take place.

If integrated at this level, the user of the container can get a huge 
benefit from
Instrumentation without ever making use of the API themselves.

> There are two fundamental issues in this equation:
>
>   * no lock in on a particular transport (i.e. transport is
>     totally configurable and released, standards based solutions
>     can be put in place)

This is already in there.  The AltRMI / HTTP / none connector system is 
entirely
configurable at runtime.   See below.  Additional connectors 
(transports) can easily
be added to the InstrumentManager code, or created externally as part of 
user
code.  The later of course does not require any modification to the IM jar.

>   * assurance that we can tie in with JMX with or without
>     full functionality

Should be no problem.  See below.

> And one open subject:
>
>   * some logic with respect to the synchronization of a management
>     model with an instrumentation model (keeping in mind that the
>     management connector model in Merlin is still in evolution)

I must apologize to having almost no knowledge of Merlin at this 
point.   I have
started taking a look at it a few times,  but under schedule crunches, I 
have always
been forced back to what I already know.  Mainly Fortress.

> Father than reinvent stuff - I would rather work with solutions that 
> exist and have been through real usage.  The counter argument is the 
> question of another dependency.  Today the instrumentation package 
> ties me into AltRMI which ties me into a legacy Avalon.  This would 
> need to be cleaned up.  Basically - AltRMI has to be removed from the 
> equation so long as it is in Incubator.

The AltRMI dependency is only at build time.  There is absolutely no 
dependency on
AltRMI if that connector is not configured in the instrument manager's 
configuration
file.   Currently, the AltRMI connector is only being used by the 
Instrument client
package.   The HTTP connector currently is capable of serving either 
human readable
HTML or XML content.  My original thought here was to replace AltRMI 
with XML
as the transport used by the instrument client.   If that was done then 
I would have no
problem removing the AltRMI connector completely.  I have not heard of 
anyone
making any other use of it.   The work is done in the HTTP connector, 
but the client
side of the job has not been started.

> Furthermore - I thing we need to come
> with the Instrumentation/JMX bridge to make this work. If this is 
> technically and politically viable - then there is a something to work 
> with.  Can you give me you opinions on this.  Just for reference - if 
> we can establish a releasable instrumentation package, I'm keen to 
> collaborate on a solution here.

The easiest way to do this would be to create a new package:
org.apache.excalibur.instrument.manager.jmx
which contained an MBean that would be registered with the instrument 
manager
as a Connector.

Connectors can be registered in the instrument manager's configuration 
file using
the following xml:
    <connectors>
        <connector class="altrmi" port="15555"/>
        <connector class="http" port="15080"/>
        <connector class="com.myco.myapp.CustomConnector" myconf="myval"/>
    </connectors>

The altrmi and http "class"s are special cases to make it easier for the 
user.  But
any class can be specified as long as it implements the 
InstrumentManagerConnector
interface.  The IM will treat them as Avalon components making
sure that they are started, stopped, configured. etc.   An instrument 
manager instance
is passed to them using the InstrumentManagerConnector.setInstrumentManager
method.  At that point, any method of the DefaultInstrumentManager class 
can be
called.

The IM will then shutdown any connectors that it initialized as part of 
its own
shutdown.

It is also possible to create and manage your own "connector" as a 
standard object.
If you give it a DefaultInstrumentManager instance then you can do with 
it as you
wish.   The connector system simply makes it easy for the user to 
configure things
within the Instrument Manager's configuration file.

Long answer, but no, I don't see any problems with being able to hook 
things into
the JMX architecture other than that I don't think it is possible to 
make all of the
information collected by the Wrapper available in an easily viewable 
format using
JMX clients.  Correct me if I am wrong.   You could definitely publish 
things like
the latest values of individual instruments etc.

As for making the Instrument and InstrumentManager packages 
releasable.   They
are very stable.  I have been using them in production environments for 
quite some
time.   The problem is the obvious lack of documentation.  As you can 
see from the
Java Service Wrapper project, writing documentation does not give me a 
rash or
anything.  But my schedule the last year has not given me any time to 
write up the
instrument docs, or do much other Avalon coding as of late.   I will try 
to get at
least the basics written up though...   It is a great tool..  But I 
can't blame anyone
for not using it without docs. :-/

Cheers,
Leif


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org

Re: instrumentation and other questions

[Stephen McConnell <mc...@apache.org>]

Leif Mortenson wrote:

> Stephen,
> At your request, moving this discussion to the dev list.

Thanks.

> As for JMX, one thing that could be done in certain situations is to 
> create an MBean for the instrument manager which would then act as 
> a JMX Connector to the InstrumentManager.

Thinking along similar lines.  In my head is the instrumentation API as 
a potential reference point concerning the instrumentation abstraction 
relative to the composition and activation frameworks.  This in turn 
could be augmented by leveraging instrumented thread and pool facilities 
in excalibur.  Using facilities, we could add in the instrumentation 
event transport solution - e.g. a JMX RMI adapter, IIOP, AltRMI or the 
existing HTTP connection facilities you have in place.

There is one subject that I want to address:

   * instrumentation of the composition and activation packages

There are two fundamental issues in this equation:

   * no lock in on a particular transport (i.e. transport is
     totally configurable and released, standards based solutions
     can be put in place)
   * assurance that we can tie in with JMX with or without
     full functionality

And one open subject:

   * some logic with respect to the synchronization of a management
     model with an instrumentation model (keeping in mind that the
     management connector model in Merlin is still in evolution)

Father than reinvent stuff - I would rather work with solutions that 
exist and have been through real usage.  The counter argument is the 
question of another dependency.  Today the instrumentation package ties 
me into AltRMI which ties me into a legacy Avalon.  This would need to 
be cleaned up.  Basically - AltRMI has to be removed from the equation 
so long as it is in Incubator.  Furthermore - I thing we need to come
with the Instrumentation/JMX bridge to make this work. If this is 
technically and politically viable - then there is a something to work 
with.  Can you give me you opinions on this.  Just for reference - if we 
can establish a releasable instrumentation package, I'm keen to 
collaborate on a solution here.

Cheers, Stephen.


> Currently AltRMI and HTTP Connectors exist.   The AltRMI Connector 
> exists which
> allows you to connect to the IM using the Swing InstrumentClient.
> 
> The HTTP connector is not quite as convenient for letting you see more 
> than one set
> of sample history at a time (that can be changed).  But it does have a 
> lot of benefits
> as well.  Including the fact that you do not need a special client to 
> connect.  Any
> browser will do.  The connector can output either HTML or XML content so it
> should be very easy to send the XML through Cocoon and transform the XML 
> into
> whatever other format you need.
> 
> Cheers,
> Leif
> 
> Stephen McConnell wrote:
> 
>> Can you do me a favor and post the following extract of your email to 
>> dev@avalon?
>>
>> My feeling is that there is a place where your work in instrumentation 
>> fits with aspects of JMX and the overall merlin architecture.  There 
>> are a bunch of thoughts I have about this but they are better on-list 
>> than off.
>>
>>
>> Leif Mortenson wrote:
>>
>>> Basically I view the Instrument package as being completely different 
>>> from JMX.
>>> As I understand it, using JVM you would be able to make individual 
>>> data points
>>> available at any given time.  But as I understand it, there is no way 
>>> to build up
>>> history or display the data points as they change over time.
>>>
>>> When I created the Instrument code.  I wanted to create a tool that 
>>> would integrated
>>> in a similar way as Logging is done.  Instruments would be added 
>>> throughout classes
>>> just as debug log output is added.  When not in use these instruments 
>>> do not affect the
>>> performance of the application.  But at any time, a client can 
>>> connect to the JVM and
>>> request a snapshot of the instrument, or ask that the instrument 
>>> start building up a
>>> history.
>>>
>>> These sample sets are requested using a lease system.   Once the 
>>> sample is requested
>>> they will continue to be collected in the JVM even if the JVM is 
>>> restarted and or the
>>> client disconnects.   It is then possible to connect at a later time, 
>>> before the lease
>>> expires, and see the instrument sample history that was built up.   
>>> Both the HTTP
>>> and instrument-client interfaces make it possible to view the sample 
>>> history as a
>>> graph.  By comparing key instrument sample sets.  It can be very easy 
>>> to tell exactly
>>> what is happening within an application.
>>>
>>> I make use of instruments for everything from pool sizes, to keeping 
>>> track of the
>>> number of times that a certain servlet is called and then how long 
>>> each call takes
>>> to complete.
>>>
>>> The instrument manager allows you to specify that certain instruments 
>>> should have
>>> permanent sample sets.  These do not require a lease and are always 
>>> available.
>>> I usually use these to track memory usage, HTTP connection and access 
>>> counts
>>> and other data which I want to see over a long period of time.
>>>
>>> Because the instrument data is preserved across JVM restarts, this 
>>> makes it
>>> possible to see how the memory usage of an application changes over 
>>> several
>>> months.
>>>
>>> I really need to get it documented so it would be easy for you try 
>>> out.  I can
>>> almost guarantee that once you understand the data that is collected 
>>> you very
>>> quickly find it as invaluable as logging.  My customers all use the 
>>> HTTP client
>>> to connect to their applications and very quickly get a feel for the 
>>> health of the
>>> app.   Memory leaks and resource problems are immediately obvious 
>>> just by
>>> glancing at the appropriate instrument sample graph.
>>
>>
>>
>>
>>
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
> For additional commands, e-mail: dev-help@avalon.apache.org
> 
> 


-- 

|------------------------------------------------|
| Magic by Merlin                                |
| Production by Avalon                           |
|                                                |
| http://avalon.apache.org/merlin                |
| http://dpml.net/merlin/distributions/latest    |
|------------------------------------------------|

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org