You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomee.apache.org by David Blevins <da...@visi.com> on 2010/05/06 18:01:37 UTC
Re: Monitoring
On Mar 4, 2010, at 12:41 PM, David Blevins wrote:
> Started looking into how we might finally get some Monitoring support into the code. Basic stats and the like viewable via JMX or something similar.
>
> Still very much in the research phase of all that. If anyone has done any work in that area, I'm interested in some tips. Or if anyone has been on the "user" side of consuming stats and has some "loves" and "hates" to share, that's also great info -- hopefully our impl would stick close to the loves and steer clear of the hates.
>
> And of course, if there are specific stats that anyone would like, definitely feel encouraged to speak up. No reason we can't put those higher in the priority list.
Been back at this one the last two weeks. Have some pretty neat code in so far.
I started with a copy of Martin Traverso's (an Apache guy) tiny jmxutil code which scans objects for @Managed annotated methods and creates a ModelMBean based on those methods. The more I started writing actual stats though, the less I liked the ModelMBean approach as it can only handle methods which then requires a lot of pointless getters just for the JMX tooling. ModelMBeans are also pretty rigid in that one object == one mbean, so it isn't possible to "roll up" attributes in to one basic MBean if those attributes are methods of different java objects.
I'm not a fan of having internal code structure exposed to users when in most cases it doesn't benefit them and simply subjects them to more brittle finer views: do users care that in my Pool object I have a "stats" object and in that object there are other "stats" objects and do they really need an individual MBean for each and every one of these fine grained objects? No, of course not. What makes my life easy as a developer, doesn't always make their lives easy as a user.
Anyway, so I wrote my own little annotation based JMX exporter. You pass in an object that has fields and/or methods annotated with @Managed. It will inspect all those fields/methods and if any of them are types that also are marked @Managed, it will walk down those types and collect its fields/methods. So it's a basic recursive tree walk, starting with the root object, that rolls up everything it finds as one flat list of JMX attributes. It also allows these fields/methods to be private in scope, so it isn't necessary to "dirty" your design to get complete stats.
The result is something exposes a nice clean view like this one:
https://issues.apache.org/jira/secure/attachment/12443866/jmx-monitoring.png
That's the view of our stateless pool stats via JConsole.
Still have a few "wires" to hookup, but it's looking good. Next part will be getting stats for each method invocation on a bean.
Comments welcome! :)
-David
Re: Monitoring
Posted by Jean-Louis MONTEIRO <je...@atosorigin.com>.
Hey David,
That's exactly what i was looking for some month ago.
I did some work on that topic too, but with that done, i gonna be able to
greatly simplify my code.
Thanks a lot for that.
Jean-Louis
stephenconnolly wrote:
>
> cool
>
> On 6 May 2010 17:01, David Blevins <da...@visi.com> wrote:
>
>>
>> On Mar 4, 2010, at 12:41 PM, David Blevins wrote:
>>
>> > Started looking into how we might finally get some Monitoring support
>> into the code. Basic stats and the like viewable via JMX or something
>> similar.
>> >
>> > Still very much in the research phase of all that. If anyone has done
>> any work in that area, I'm interested in some tips. Or if anyone has
>> been
>> on the "user" side of consuming stats and has some "loves" and "hates" to
>> share, that's also great info -- hopefully our impl would stick close to
>> the
>> loves and steer clear of the hates.
>> >
>> > And of course, if there are specific stats that anyone would like,
>> definitely feel encouraged to speak up. No reason we can't put those
>> higher
>> in the priority list.
>>
>> Been back at this one the last two weeks. Have some pretty neat code in
>> so
>> far.
>>
>> I started with a copy of Martin Traverso's (an Apache guy) tiny jmxutil
>> code which scans objects for @Managed annotated methods and creates a
>> ModelMBean based on those methods. The more I started writing actual
>> stats
>> though, the less I liked the ModelMBean approach as it can only handle
>> methods which then requires a lot of pointless getters just for the JMX
>> tooling. ModelMBeans are also pretty rigid in that one object == one
>> mbean,
>> so it isn't possible to "roll up" attributes in to one basic MBean if
>> those
>> attributes are methods of different java objects.
>>
>> I'm not a fan of having internal code structure exposed to users when in
>> most cases it doesn't benefit them and simply subjects them to more
>> brittle
>> finer views: do users care that in my Pool object I have a "stats" object
>> and in that object there are other "stats" objects and do they really
>> need
>> an individual MBean for each and every one of these fine grained objects?
>> No, of course not. What makes my life easy as a developer, doesn't
>> always
>> make their lives easy as a user.
>>
>> Anyway, so I wrote my own little annotation based JMX exporter. You pass
>> in an object that has fields and/or methods annotated with @Managed. It
>> will inspect all those fields/methods and if any of them are types that
>> also
>> are marked @Managed, it will walk down those types and collect its
>> fields/methods. So it's a basic recursive tree walk, starting with the
>> root
>> object, that rolls up everything it finds as one flat list of JMX
>> attributes. It also allows these fields/methods to be private in scope,
>> so
>> it isn't necessary to "dirty" your design to get complete stats.
>>
>> The result is something exposes a nice clean view like this one:
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12443866/jmx-monitoring.png
>>
>> That's the view of our stateless pool stats via JConsole.
>>
>> Still have a few "wires" to hookup, but it's looking good. Next part
>> will
>> be getting stats for each method invocation on a bean.
>>
>> Comments welcome! :)
>>
>>
>> -David
>>
>>
>>
>>
>
>
--
View this message in context: http://openejb.979440.n4.nabble.com/Re-Monitoring-tp2132887p2133789.html
Sent from the OpenEJB Dev mailing list archive at Nabble.com.
Fwd: Monitoring - MBean names?
Posted by David Blevins <da...@visi.com>.
Trying to collect as much feedback as I can. See below.
Here's the nabble link to the thread for those who aren't currently subscribed:
http://openejb.979440.n4.nabble.com/Re-Monitoring-tp2132887p2173234.html
-David
Begin forwarded message:
> Resent-From: <db...@visi.com>
> From: David Blevins <da...@visi.com>
> Date: May 10, 2010 7:44:36 PM PDT
> To: dev@openejb.apache.org
> Subject: Re: Monitoring - MBean names?
> Reply-To: dev@openejb.apache.org
>
> Still largely uneducated when it comes to what is a good ObjectName and very much looking for feedback.
>
> Currently, we'll have three MBeans for each @Stateless bean, one to represent the "Pool", one for "Invocations", and one just generic for the bean in general (still hacking that one up).
>
> What should the ObjectName for each of these be?
>
>
> -David
>
>
Re: Monitoring - MBean names?
Posted by David Jencks <da...@yahoo.com>.
jsr77 specifies the name for the mbean representing the ejb itself. Except for the domain name part, its the same as the map part of the geronimo gbean name for the ejb gbean. Jsr77 is not my strong point but I thought there were supposed to be some statistics possibly available from the mbean for the ejb. You might want to make your mbeans "children" of the jsr77 ejb mbean,
There's a method for making a child abstract name in geronimo like this (from JSr77Naming.java in framework/modules/geronimo-kernel):
public AbstractName createChildName(AbstractName parentAbstractName, Artifact artifact, String name, String type) {
if (name == null) {
throw new NullPointerException("No name supplied");
}
if (type == null) {
throw new NullPointerException("No type supplied");
}
Map nameMap = new HashMap(parentAbstractName.getName());
String parentType = (String) nameMap.remove(J2EE_TYPE);
String parentName = (String) nameMap.remove(J2EE_NAME);
if (parentType == null) {
throw new IllegalArgumentException("parent name must have a j2eeType name component");
}
if (INVALID_GENERIC_PARENT_TYPE.equals(parentType)) {
throw new IllegalArgumentException("You can't create a child of a generic typed gbean");
}
nameMap.put(parentType, parentName);
nameMap.put(J2EE_TYPE, type);
nameMap.put(J2EE_NAME, name);
return new AbstractName(artifact,
nameMap,
createObjectName(nameMap));
}
The crucial point here is the parent's last name component is name=foo, and this has to be turned into parentType=foo,name=foo-child.
I'm not sure what will happen in geronimo when all the gbeans are exposed in jmx and you add some more not backed by gbeans. However, this is something we'll need to figure out.
thanks
david jencks
On May 10, 2010, at 7:44 PM, David Blevins wrote:
> Still largely uneducated when it comes to what is a good ObjectName and very much looking for feedback.
>
> Currently, we'll have three MBeans for each @Stateless bean, one to represent the "Pool", one for "Invocations", and one just generic for the bean in general (still hacking that one up).
>
> What should the ObjectName for each of these be?
>
>
> -David
>
Re: Monitoring - MBean names?
Posted by David Blevins <da...@visi.com>.
Still largely uneducated when it comes to what is a good ObjectName and very much looking for feedback.
Currently, we'll have three MBeans for each @Stateless bean, one to represent the "Pool", one for "Invocations", and one just generic for the bean in general (still hacking that one up).
What should the ObjectName for each of these be?
-David
Re: Monitoring
Posted by David Blevins <da...@visi.com>.
On May 9, 2010, at 10:52 PM, Stephen Connolly wrote:
> I'm guessing your monitoring code is in a nice separate module that could be
> used outside of openejb for monitoring anyway?
It's currently in the openejb-core module with one class "@Monitor" in the openejb-api module. It has a dep on commons-math at the moment, but I may just pull in the few classes we use and ditch the 1.5 MB jar.
-David
>
> On 10 May 2010 02:26, David Blevins <da...@visi.com> wrote:
>
>>
>> On May 6, 2010, at 9:01 AM, David Blevins wrote:
>>
>>> The result is something exposes a nice clean view like this one:
>>>
>>>
>> https://issues.apache.org/jira/secure/attachment/12443866/jmx-monitoring.png
>>>
>>> That's the view of our stateless pool stats via JConsole.
>>>
>>> Still have a few "wires" to hookup, but it's looking good. Next part
>> will be getting stats for each method invocation on a bean.
>>
>> Bean method stats are mostly in (same as above, need to hook them in). As
>> monitoring methods can get expensive, I added an @Monitor annotation that
>> allows people to annotate the methods of their bean to elect which should be
>> monitored and how much of a sample size to keep for each method. The
>> annotation can be used at the class level as well to enable monitoring for
>> all methods of the bean.
>>
>> So here's the view of a fully monitored class:
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12444073/jmx-invocations-all-stats.png
>>
>> As that can be a lot to chew on and I am a grep addict, I thought it might
>> be nice to offer a standard operation on all our MBeans that allow you to
>> filter (essentially grep) the MBean attributes. Here's an example of it
>> being used to filter out all attributes except those for the "red()" method:
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12444074/jmx-invocations-filter-a1.png
>>
>> And the resulting MBean attributes, post filtering. With JConsole at
>> least, it seems you have to disconnect and reconnect to force it to reload
>> the MBean definition.
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12444075/jmx-invocations-filter-a2.png
>>
>> Here's another example if it being used to filter out all but the "Count"
>> attributes, which allows us to see which methods are invoked the most.
>>
>>
>> https://issues.apache.org/jira/secure/attachment/12444076/jmx-invocations-filter-b1.png
>>
>> https://issues.apache.org/jira/secure/attachment/12444077/jmx-invocations-filter-b2.png
>>
>>
>> -David
>>
>>
Re: Monitoring
Posted by Stephen Connolly <st...@gmail.com>.
I'm guessing your monitoring code is in a nice separate module that could be
used outside of openejb for monitoring anyway?
On 10 May 2010 02:26, David Blevins <da...@visi.com> wrote:
>
> On May 6, 2010, at 9:01 AM, David Blevins wrote:
>
> > The result is something exposes a nice clean view like this one:
> >
> >
> https://issues.apache.org/jira/secure/attachment/12443866/jmx-monitoring.png
> >
> > That's the view of our stateless pool stats via JConsole.
> >
> > Still have a few "wires" to hookup, but it's looking good. Next part
> will be getting stats for each method invocation on a bean.
>
> Bean method stats are mostly in (same as above, need to hook them in). As
> monitoring methods can get expensive, I added an @Monitor annotation that
> allows people to annotate the methods of their bean to elect which should be
> monitored and how much of a sample size to keep for each method. The
> annotation can be used at the class level as well to enable monitoring for
> all methods of the bean.
>
> So here's the view of a fully monitored class:
>
>
> https://issues.apache.org/jira/secure/attachment/12444073/jmx-invocations-all-stats.png
>
> As that can be a lot to chew on and I am a grep addict, I thought it might
> be nice to offer a standard operation on all our MBeans that allow you to
> filter (essentially grep) the MBean attributes. Here's an example of it
> being used to filter out all attributes except those for the "red()" method:
>
>
> https://issues.apache.org/jira/secure/attachment/12444074/jmx-invocations-filter-a1.png
>
> And the resulting MBean attributes, post filtering. With JConsole at
> least, it seems you have to disconnect and reconnect to force it to reload
> the MBean definition.
>
>
> https://issues.apache.org/jira/secure/attachment/12444075/jmx-invocations-filter-a2.png
>
> Here's another example if it being used to filter out all but the "Count"
> attributes, which allows us to see which methods are invoked the most.
>
>
> https://issues.apache.org/jira/secure/attachment/12444076/jmx-invocations-filter-b1.png
>
> https://issues.apache.org/jira/secure/attachment/12444077/jmx-invocations-filter-b2.png
>
>
> -David
>
>
Re: Monitoring
Posted by David Blevins <da...@visi.com>.
On May 6, 2010, at 9:01 AM, David Blevins wrote:
> The result is something exposes a nice clean view like this one:
>
> https://issues.apache.org/jira/secure/attachment/12443866/jmx-monitoring.png
>
> That's the view of our stateless pool stats via JConsole.
>
> Still have a few "wires" to hookup, but it's looking good. Next part will be getting stats for each method invocation on a bean.
Bean method stats are mostly in (same as above, need to hook them in). As monitoring methods can get expensive, I added an @Monitor annotation that allows people to annotate the methods of their bean to elect which should be monitored and how much of a sample size to keep for each method. The annotation can be used at the class level as well to enable monitoring for all methods of the bean.
So here's the view of a fully monitored class:
https://issues.apache.org/jira/secure/attachment/12444073/jmx-invocations-all-stats.png
As that can be a lot to chew on and I am a grep addict, I thought it might be nice to offer a standard operation on all our MBeans that allow you to filter (essentially grep) the MBean attributes. Here's an example of it being used to filter out all attributes except those for the "red()" method:
https://issues.apache.org/jira/secure/attachment/12444074/jmx-invocations-filter-a1.png
And the resulting MBean attributes, post filtering. With JConsole at least, it seems you have to disconnect and reconnect to force it to reload the MBean definition.
https://issues.apache.org/jira/secure/attachment/12444075/jmx-invocations-filter-a2.png
Here's another example if it being used to filter out all but the "Count" attributes, which allows us to see which methods are invoked the most.
https://issues.apache.org/jira/secure/attachment/12444076/jmx-invocations-filter-b1.png
https://issues.apache.org/jira/secure/attachment/12444077/jmx-invocations-filter-b2.png
-David
Re: Monitoring
Posted by Stephen Connolly <st...@gmail.com>.
cool
On 6 May 2010 17:01, David Blevins <da...@visi.com> wrote:
>
> On Mar 4, 2010, at 12:41 PM, David Blevins wrote:
>
> > Started looking into how we might finally get some Monitoring support
> into the code. Basic stats and the like viewable via JMX or something
> similar.
> >
> > Still very much in the research phase of all that. If anyone has done
> any work in that area, I'm interested in some tips. Or if anyone has been
> on the "user" side of consuming stats and has some "loves" and "hates" to
> share, that's also great info -- hopefully our impl would stick close to the
> loves and steer clear of the hates.
> >
> > And of course, if there are specific stats that anyone would like,
> definitely feel encouraged to speak up. No reason we can't put those higher
> in the priority list.
>
> Been back at this one the last two weeks. Have some pretty neat code in so
> far.
>
> I started with a copy of Martin Traverso's (an Apache guy) tiny jmxutil
> code which scans objects for @Managed annotated methods and creates a
> ModelMBean based on those methods. The more I started writing actual stats
> though, the less I liked the ModelMBean approach as it can only handle
> methods which then requires a lot of pointless getters just for the JMX
> tooling. ModelMBeans are also pretty rigid in that one object == one mbean,
> so it isn't possible to "roll up" attributes in to one basic MBean if those
> attributes are methods of different java objects.
>
> I'm not a fan of having internal code structure exposed to users when in
> most cases it doesn't benefit them and simply subjects them to more brittle
> finer views: do users care that in my Pool object I have a "stats" object
> and in that object there are other "stats" objects and do they really need
> an individual MBean for each and every one of these fine grained objects?
> No, of course not. What makes my life easy as a developer, doesn't always
> make their lives easy as a user.
>
> Anyway, so I wrote my own little annotation based JMX exporter. You pass
> in an object that has fields and/or methods annotated with @Managed. It
> will inspect all those fields/methods and if any of them are types that also
> are marked @Managed, it will walk down those types and collect its
> fields/methods. So it's a basic recursive tree walk, starting with the root
> object, that rolls up everything it finds as one flat list of JMX
> attributes. It also allows these fields/methods to be private in scope, so
> it isn't necessary to "dirty" your design to get complete stats.
>
> The result is something exposes a nice clean view like this one:
>
>
> https://issues.apache.org/jira/secure/attachment/12443866/jmx-monitoring.png
>
> That's the view of our stateless pool stats via JConsole.
>
> Still have a few "wires" to hookup, but it's looking good. Next part will
> be getting stats for each method invocation on a bean.
>
> Comments welcome! :)
>
>
> -David
>
>
>
>