You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Matt Hogstrom <ma...@hogstrom.org> on 2007/06/14 15:08:53 UTC
[DISCUSS] Improving diagnostics
Lately I've been working with users in debugging various application
problems. Some of the problems are merely configuration but others
are deeper application / infrastructure problems. Regardless of the
type of problem I've never personally been satisfied with the
diagnostic information produced by the server (this isn't a Geronimo
statement but really AppServers as a whole including WebSphere and
WebLogic).
Here are some thoughts that I want to pursue:
I’ve been working with some customers lately and the work has
centered around debugging some of the aspects of their server. In
this case it was using Apache Geronimo but the problem really applies
to most application servers in general. For the most part there is
little diagnostic information available when an application fails.
We get the ever popular nested Java Stack trace which is certainly a
good indicator of where a failure occurred but is woefully inadequate
in many instances of why a failure occurred. This get’s worse in
that for the most part people need to recreate the problem with
additional tracing and, in the worst case, additional diagnostic code
in their application. Wouldn’t it be nice if some of the diagnostic
capability that was needed was included in the server itself?
Over the next few weeks I’m going to be doing some experimentation on
how to improve server diagnostics through the use of Aspects and/or
Instrumentation. Since this is experimental we’ll see what the final
result will be. Here are some of my initial goals:
1. Improve diagnostics by providing a Diagnostic Report when a
Tx fails.
2. Provide better visualization of Java Stack traces so problem
areas pop out.
3. Capture wait information
For number 1 I’m going to focus on servlets to begin with given that
they represent the preponderance of requests made in AppServers
today. This information will include information from the request
object, the servlet being invoked, invocation time, transaction ID
(if it exists), enlisted connections (database and messaging), oh
yeah, and the Thread ID of execution. This is a mouthful to begin
with anyway.
Number 2 is really just applying some template information on a Java
Stack trace. I want application classes to standout so developers
will be able to quickly see where their application is involved.
Infrastructure pieces like the server, Hibernate, TopLink, etc. would
also be highlighted in a different color and style and plain old java
classes would be a boring style as their are merely pawns in the
transactional game.
Finally, wouldn’t it be nice to know how long a thread has been
waiting and for what reason? Is it waiting on a request from another
server or perhaps there is a database locking problem. Did a
WebService go awry? Basically, I want to know what in the heck a
thread is waiting on.
Please chime in with your thoughts. I spect that Aspects or
Instrumentation may be the only way to go for some of this as many of
the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
Posted by Matt Hogstrom <ma...@hogstrom.org>.
On Jun 14, 2007, at 10:51 AM, Paul McMahan wrote:
> I like your ideas! Thanks for bringing up this important topic.
> One of my pet peeves in debugging geronimo apps is when I'm trying
> to step through a gbean call but the debugger doesn't have the
> source available since cglib has dynamically generated the proxy
> classes. I usually end up having to do cheesy things like setting
> breakpoints on both sides of the proxy, which only works if I can
> figure out what's on the other side of the proxy. Seems like I
> remember some talk a while ago about an experimental runtime mode
> which disables proxying. I don't think you would want to normally
> run in that mode but it would be useful for debugging purposes. I
> don't remember where that ended up.
Start the server with the additional option:
-DXorg.apache.geronimo.gbean.NoProxy=true
If you have time I think the only reason that inhibits us from
running the server in this mode is that there are a few Portlets that
depend on Proxies. It would be excellent to remove that dependency.
Re: [DISCUSS] Improving diagnostics
Posted by Jeff Genender <jg...@apache.org>.
Paul,
I believe there is a setting you can use to turn off the proxies and not
have to deal with CGLib...perfect for debugging (I think Dain pu t it in).
Jeff
Paul McMahan wrote:
> I like your ideas! Thanks for bringing up this important topic. One of
> my pet peeves in debugging geronimo apps is when I'm trying to step
> through a gbean call but the debugger doesn't have the source available
> since cglib has dynamically generated the proxy classes. I usually end
> up having to do cheesy things like setting breakpoints on both sides of
> the proxy, which only works if I can figure out what's on the other side
> of the proxy. Seems like I remember some talk a while ago about an
> experimental runtime mode which disables proxying. I don't think you
> would want to normally run in that mode but it would be useful for
> debugging purposes. I don't remember where that ended up. One other
> thing that I think would be very helpful for debug and diagnostics would
> be to serialize gbean info to xml rather than a serialized java
> object. Again, I think there was some experimental work on this a few
> months ago but where did it end up? Maybe these improvements are
> already and I just didn't know about it :-)
>
> Best wishes,
> Paul
>
> On Jun 14, 2007, at 9:08 AM, Matt Hogstrom wrote:
>
>> Lately I've been working with users in debugging various application
>> problems. Some of the problems are merely configuration but others
>> are deeper application / infrastructure problems. Regardless of the
>> type of problem I've never personally been satisfied with the
>> diagnostic information produced by the server (this isn't a Geronimo
>> statement but really AppServers as a whole including WebSphere and
>> WebLogic).
>>
>> Here are some thoughts that I want to pursue:
>>
>> I’ve been working with some customers lately and the work has centered
>> around debugging some of the aspects of their server. In this case it
>> was using Apache Geronimo but the problem really applies to most
>> application servers in general. For the most part there is little
>> diagnostic information available when an application fails. We get
>> the ever popular nested Java Stack trace which is certainly a good
>> indicator of where a failure occurred but is woefully inadequate in
>> many instances of why a failure occurred. This get’s worse in that
>> for the most part people need to recreate the problem with additional
>> tracing and, in the worst case, additional diagnostic code in their
>> application. Wouldn’t it be nice if some of the diagnostic capability
>> that was needed was included in the server itself?
>>
>> Over the next few weeks I’m going to be doing some experimentation on
>> how to improve server diagnostics through the use of Aspects and/or
>> Instrumentation. Since this is experimental we’ll see what the final
>> result will be. Here are some of my initial goals:
>>
>> 1. Improve diagnostics by providing a Diagnostic Report when a Tx
>> fails.
>> 2. Provide better visualization of Java Stack traces so problem
>> areas pop out.
>> 3. Capture wait information
>>
>> For number 1 I’m going to focus on servlets to begin with given that
>> they represent the preponderance of requests made in AppServers
>> today. This information will include information from the request
>> object, the servlet being invoked, invocation time, transaction ID (if
>> it exists), enlisted connections (database and messaging), oh yeah,
>> and the Thread ID of execution. This is a mouthful to begin with anyway.
>>
>> Number 2 is really just applying some template information on a Java
>> Stack trace. I want application classes to standout so developers
>> will be able to quickly see where their application is involved.
>> Infrastructure pieces like the server, Hibernate, TopLink, etc. would
>> also be highlighted in a different color and style and plain old java
>> classes would be a boring style as their are merely pawns in the
>> transactional game.
>>
>> Finally, wouldn’t it be nice to know how long a thread has been
>> waiting and for what reason? Is it waiting on a request from another
>> server or perhaps there is a database locking problem. Did a
>> WebService go awry? Basically, I want to know what in the heck a
>> thread is waiting on.
>>
>> Please chime in with your thoughts. I spect that Aspects or
>> Instrumentation may be the only way to go for some of this as many of
>> the components we include won't have this capability in them.
>
Re: [DISCUSS] Improving diagnostics
Posted by "Jay D. McHugh" <ja...@joyfulnoisewebdesign.com>.
Is there anyway we could intercept when exceptions are created? If we
could then one possibility to improve diagnostics would be to keep a
count of what exceptions are being thrown and from where. With that
information, we could perhaps give an administrator hints on where
trouble spots are in their applications. Or, we could automatically
escalate the level of logging/information provided in areas that throw
lots of exceptions (subject to some user configured threshold).
And regarding XML serialization - Does anyone know where the XML
serialization work was done or who did it (if there really was any work
on this)?
I am trying to rework some of the console screens to get better browser
compatibility and sending information back and forth in XML will prevent
some javascript security problems.
If someone has already figured out the schema to describe GBeans then
that would make my work on the console easier too.
Jay
Paul McMahan wrote:
> I like your ideas! Thanks for bringing up this important topic. One
> of my pet peeves in debugging geronimo apps is when I'm trying to step
> through a gbean call but the debugger doesn't have the source
> available since cglib has dynamically generated the proxy classes. I
> usually end up having to do cheesy things like setting breakpoints on
> both sides of the proxy, which only works if I can figure out what's
> on the other side of the proxy. Seems like I remember some talk a
> while ago about an experimental runtime mode which disables proxying.
> I don't think you would want to normally run in that mode but it would
> be useful for debugging purposes. I don't remember where that ended
> up. One other thing that I think would be very helpful for debug and
> diagnostics would be to serialize gbean info to xml rather than a
> serialized java object. Again, I think there was some experimental
> work on this a few months ago but where did it end up? Maybe these
> improvements are already and I just didn't know about it :-)
>
> Best wishes,
> Paul
>
> On Jun 14, 2007, at 9:08 AM, Matt Hogstrom wrote:
>
>> Lately I've been working with users in debugging various application
>> problems. Some of the problems are merely configuration but others
>> are deeper application / infrastructure problems. Regardless of the
>> type of problem I've never personally been satisfied with the
>> diagnostic information produced by the server (this isn't a Geronimo
>> statement but really AppServers as a whole including WebSphere and
>> WebLogic).
>>
>> Here are some thoughts that I want to pursue:
>>
>> I’ve been working with some customers lately and the work has
>> centered around debugging some of the aspects of their server. In
>> this case it was using Apache Geronimo but the problem really applies
>> to most application servers in general. For the most part there is
>> little diagnostic information available when an application fails.
>> We get the ever popular nested Java Stack trace which is certainly a
>> good indicator of where a failure occurred but is woefully inadequate
>> in many instances of why a failure occurred. This get’s worse in
>> that for the most part people need to recreate the problem with
>> additional tracing and, in the worst case, additional diagnostic code
>> in their application. Wouldn’t it be nice if some of the diagnostic
>> capability that was needed was included in the server itself?
>>
>> Over the next few weeks I’m going to be doing some experimentation on
>> how to improve server diagnostics through the use of Aspects and/or
>> Instrumentation. Since this is experimental we’ll see what the final
>> result will be. Here are some of my initial goals:
>>
>> 1. Improve diagnostics by providing a Diagnostic Report when a Tx
>> fails.
>> 2. Provide better visualization of Java Stack traces so problem
>> areas pop out.
>> 3. Capture wait information
>>
>> For number 1 I’m going to focus on servlets to begin with given that
>> they represent the preponderance of requests made in AppServers
>> today. This information will include information from the request
>> object, the servlet being invoked, invocation time, transaction ID
>> (if it exists), enlisted connections (database and messaging), oh
>> yeah, and the Thread ID of execution. This is a mouthful to begin
>> with anyway.
>>
>> Number 2 is really just applying some template information on a Java
>> Stack trace. I want application classes to standout so developers
>> will be able to quickly see where their application is involved.
>> Infrastructure pieces like the server, Hibernate, TopLink, etc. would
>> also be highlighted in a different color and style and plain old java
>> classes would be a boring style as their are merely pawns in the
>> transactional game.
>>
>> Finally, wouldn’t it be nice to know how long a thread has been
>> waiting and for what reason? Is it waiting on a request from another
>> server or perhaps there is a database locking problem. Did a
>> WebService go awry? Basically, I want to know what in the heck a
>> thread is waiting on.
>>
>> Please chime in with your thoughts. I spect that Aspects or
>> Instrumentation may be the only way to go for some of this as many of
>> the components we include won't have this capability in them.
>
>
>
>
Re: [DISCUSS] Improving diagnostics
Posted by Paul McMahan <pa...@gmail.com>.
I like your ideas! Thanks for bringing up this important topic. One
of my pet peeves in debugging geronimo apps is when I'm trying to
step through a gbean call but the debugger doesn't have the source
available since cglib has dynamically generated the proxy classes. I
usually end up having to do cheesy things like setting breakpoints on
both sides of the proxy, which only works if I can figure out what's
on the other side of the proxy. Seems like I remember some talk a
while ago about an experimental runtime mode which disables
proxying. I don't think you would want to normally run in that mode
but it would be useful for debugging purposes. I don't remember
where that ended up. One other thing that I think would be very
helpful for debug and diagnostics would be to serialize gbean info to
xml rather than a serialized java object. Again, I think there was
some experimental work on this a few months ago but where did it end
up? Maybe these improvements are already and I just didn't know
about it :-)
Best wishes,
Paul
On Jun 14, 2007, at 9:08 AM, Matt Hogstrom wrote:
> Lately I've been working with users in debugging various
> application problems. Some of the problems are merely
> configuration but others are deeper application / infrastructure
> problems. Regardless of the type of problem I've never personally
> been satisfied with the diagnostic information produced by the
> server (this isn't a Geronimo statement but really AppServers as a
> whole including WebSphere and WebLogic).
>
> Here are some thoughts that I want to pursue:
>
> I’ve been working with some customers lately and the work has
> centered around debugging some of the aspects of their server. In
> this case it was using Apache Geronimo but the problem really
> applies to most application servers in general. For the most part
> there is little diagnostic information available when an
> application fails. We get the ever popular nested Java Stack trace
> which is certainly a good indicator of where a failure occurred but
> is woefully inadequate in many instances of why a failure
> occurred. This get’s worse in that for the most part people need
> to recreate the problem with additional tracing and, in the worst
> case, additional diagnostic code in their application. Wouldn’t it
> be nice if some of the diagnostic capability that was needed was
> included in the server itself?
>
> Over the next few weeks I’m going to be doing some experimentation
> on how to improve server diagnostics through the use of Aspects and/
> or Instrumentation. Since this is experimental we’ll see what the
> final result will be. Here are some of my initial goals:
>
> 1. Improve diagnostics by providing a Diagnostic Report when a
> Tx fails.
> 2. Provide better visualization of Java Stack traces so problem
> areas pop out.
> 3. Capture wait information
>
> For number 1 I’m going to focus on servlets to begin with given
> that they represent the preponderance of requests made in
> AppServers today. This information will include information from
> the request object, the servlet being invoked, invocation time,
> transaction ID (if it exists), enlisted connections (database and
> messaging), oh yeah, and the Thread ID of execution. This is a
> mouthful to begin with anyway.
>
> Number 2 is really just applying some template information on a
> Java Stack trace. I want application classes to standout so
> developers will be able to quickly see where their application is
> involved. Infrastructure pieces like the server, Hibernate,
> TopLink, etc. would also be highlighted in a different color and
> style and plain old java classes would be a boring style as their
> are merely pawns in the transactional game.
>
> Finally, wouldn’t it be nice to know how long a thread has been
> waiting and for what reason? Is it waiting on a request from
> another server or perhaps there is a database locking problem. Did
> a WebService go awry? Basically, I want to know what in the heck a
> thread is waiting on.
>
> Please chime in with your thoughts. I spect that Aspects or
> Instrumentation may be the only way to go for some of this as many
> of the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
Posted by Lin Sun <li...@gmail.com>.
I like all your ideas! One thing I have been spending some time is to
how to turn on different levels of logs at different projects (inside of
core geronimo) or other geronimo dependent logs. It would be very nice
if the console can provide that level of details so that a user can just
turn on the logging for the few components they need. Even better, the
system is smart to figure out which components might be interesting for
the specific error, and turn on the logs for the user after the user
requests it via a one click (instead of multiple clicks after figuring
out which components they need to turn on).
Lin
Matt Hogstrom wrote:
> Lately I've been working with users in debugging various application
> problems. Some of the problems are merely configuration but others are
> deeper application / infrastructure problems. Regardless of the type of
> problem I've never personally been satisfied with the diagnostic
> information produced by the server (this isn't a Geronimo statement but
> really AppServers as a whole including WebSphere and WebLogic).
>
> Here are some thoughts that I want to pursue:
>
> I’ve been working with some customers lately and the work has centered
> around debugging some of the aspects of their server. In this case it
> was using Apache Geronimo but the problem really applies to most
> application servers in general. For the most part there is little
> diagnostic information available when an application fails. We get the
> ever popular nested Java Stack trace which is certainly a good indicator
> of where a failure occurred but is woefully inadequate in many instances
> of why a failure occurred. This get’s worse in that for the most part
> people need to recreate the problem with additional tracing and, in the
> worst case, additional diagnostic code in their application. Wouldn’t
> it be nice if some of the diagnostic capability that was needed was
> included in the server itself?
>
> Over the next few weeks I’m going to be doing some experimentation on
> how to improve server diagnostics through the use of Aspects and/or
> Instrumentation. Since this is experimental we’ll see what the final
> result will be. Here are some of my initial goals:
>
> 1. Improve diagnostics by providing a Diagnostic Report when a Tx
> fails.
> 2. Provide better visualization of Java Stack traces so problem areas
> pop out.
> 3. Capture wait information
>
> For number 1 I’m going to focus on servlets to begin with given that
> they represent the preponderance of requests made in AppServers today.
> This information will include information from the request object, the
> servlet being invoked, invocation time, transaction ID (if it exists),
> enlisted connections (database and messaging), oh yeah, and the Thread
> ID of execution. This is a mouthful to begin with anyway.
>
> Number 2 is really just applying some template information on a Java
> Stack trace. I want application classes to standout so developers will
> be able to quickly see where their application is involved.
> Infrastructure pieces like the server, Hibernate, TopLink, etc. would
> also be highlighted in a different color and style and plain old java
> classes would be a boring style as their are merely pawns in the
> transactional game.
>
> Finally, wouldn’t it be nice to know how long a thread has been waiting
> and for what reason? Is it waiting on a request from another server or
> perhaps there is a database locking problem. Did a WebService go awry?
> Basically, I want to know what in the heck a thread is waiting on.
>
> Please chime in with your thoughts. I spect that Aspects or
> Instrumentation may be the only way to go for some of this as many of
> the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
Posted by Jason Dillon <ja...@planet57.com>.
Um... sure... but now right now... too many beers.
:-P
But JCL or SLF4J, we still need some serious logging reform...
--jason
On Jun 15, 2007, at 6:36 AM, Jacek Laskowski wrote:
> On 6/15/07, Jason Dillon <ja...@planet57.com> wrote:
>
>> And, I think we should seriously consider using SLF4J ( http://
>> www.slf4j.org/ ) and drop JCL.
>
> I've never used SLF4J before, but it's again I could hear about it
> from you (I'm pretty sure it was you who tried to persuade us to use
> it). Could you tell a bit more why slf4j is better than jcl. Why is it
> worth to move to slf4j - I guess it'd mean a lot of changes in the
> code, wouldn't it?
>
> Jacek
>
> --
> Jacek Laskowski
> http://www.JacekLaskowski.pl
Re: [DISCUSS] Improving diagnostics
Posted by Jacek Laskowski <ja...@laskowski.net.pl>.
On 6/15/07, Jason Dillon <ja...@planet57.com> wrote:
> And, I think we should seriously consider using SLF4J ( http://
> www.slf4j.org/ ) and drop JCL.
I've never used SLF4J before, but it's again I could hear about it
from you (I'm pretty sure it was you who tried to persuade us to use
it). Could you tell a bit more why slf4j is better than jcl. Why is it
worth to move to slf4j - I guess it'd mean a lot of changes in the
code, wouldn't it?
Jacek
--
Jacek Laskowski
http://www.JacekLaskowski.pl
Re: [DISCUSS] Improving diagnostics
Posted by Kevan Miller <ke...@gmail.com>.
On Jun 15, 2007, at 3:24 AM, Jason Dillon wrote:
> I think that in general we need to go through some serious logging
> reform. IMO logging is critical and right now I think the state of
> logging in G is a mess. I would like to see more effort put into
> making logging more consistent and provide better user tools to
> control the servers logging as well as application logging.
I totally agree. Was thinking the same thing last night... IMO,
fixing this problem goes a long way towards addressing Matt's
concerns... In most cases, it's *lack* of information that inhibits
diagnosis, not how the information is presented...
Some general architecture/standardization is needed on logging format
(including Thread ids, etc -- start to address some of Matt's
concerns). Followed by a concerted effort to redo/introduce logging
into the majority of Geronimo's components.
>
> And, I think we should seriously consider using SLF4J ( http://
> www.slf4j.org/ ) and drop JCL.
Can't say that I'm super motivated to move off of JCL, but I'm not
averse to it, either...
--kevan
Re: [DISCUSS] Improving diagnostics
Posted by Jason Dillon <ja...@planet57.com>.
I think that in general we need to go through some serious logging
reform. IMO logging is critical and right now I think the state of
logging in G is a mess. I would like to see more effort put into
making logging more consistent and provide better user tools to
control the servers logging as well as application logging.
And, I think we should seriously consider using SLF4J ( http://
www.slf4j.org/ ) and drop JCL.
--jason
On Jun 14, 2007, at 6:08 AM, Matt Hogstrom wrote:
> Lately I've been working with users in debugging various
> application problems. Some of the problems are merely
> configuration but others are deeper application / infrastructure
> problems. Regardless of the type of problem I've never personally
> been satisfied with the diagnostic information produced by the
> server (this isn't a Geronimo statement but really AppServers as a
> whole including WebSphere and WebLogic).
>
> Here are some thoughts that I want to pursue:
>
> I’ve been working with some customers lately and the work has
> centered around debugging some of the aspects of their server. In
> this case it was using Apache Geronimo but the problem really
> applies to most application servers in general. For the most part
> there is little diagnostic information available when an
> application fails. We get the ever popular nested Java Stack trace
> which is certainly a good indicator of where a failure occurred but
> is woefully inadequate in many instances of why a failure
> occurred. This get’s worse in that for the most part people need
> to recreate the problem with additional tracing and, in the worst
> case, additional diagnostic code in their application. Wouldn’t it
> be nice if some of the diagnostic capability that was needed was
> included in the server itself?
>
> Over the next few weeks I’m going to be doing some experimentation
> on how to improve server diagnostics through the use of Aspects and/
> or Instrumentation. Since this is experimental we’ll see what the
> final result will be. Here are some of my initial goals:
>
> 1. Improve diagnostics by providing a Diagnostic Report when a
> Tx fails.
> 2. Provide better visualization of Java Stack traces so problem
> areas pop out.
> 3. Capture wait information
>
> For number 1 I’m going to focus on servlets to begin with given
> that they represent the preponderance of requests made in
> AppServers today. This information will include information from
> the request object, the servlet being invoked, invocation time,
> transaction ID (if it exists), enlisted connections (database and
> messaging), oh yeah, and the Thread ID of execution. This is a
> mouthful to begin with anyway.
>
> Number 2 is really just applying some template information on a
> Java Stack trace. I want application classes to standout so
> developers will be able to quickly see where their application is
> involved. Infrastructure pieces like the server, Hibernate,
> TopLink, etc. would also be highlighted in a different color and
> style and plain old java classes would be a boring style as their
> are merely pawns in the transactional game.
>
> Finally, wouldn’t it be nice to know how long a thread has been
> waiting and for what reason? Is it waiting on a request from
> another server or perhaps there is a database locking problem. Did
> a WebService go awry? Basically, I want to know what in the heck a
> thread is waiting on.
>
> Please chime in with your thoughts. I spect that Aspects or
> Instrumentation may be the only way to go for some of this as many
> of the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
Posted by Matt Hogstrom <ma...@hogstrom.org>.
I saw the work you did in OEJB. This would be a HUGE help as well.
On Jun 14, 2007, at 9:44 AM, Rick McGuire wrote:
> One thing that might be a useful first start would be something
> similar to the "Hungry Exception" cleanup effort that was done with
> openejb3. Basically, this just ensured that any place where an
> exception gets thrown because of another caught exception that the
> original exception was maintained as a cause. I've spend a lot of
> time grumbling about situations where the original exception
> information was thrown away, sometimes through multiple levels of
> failure.
>
> Rick
>
>
> Matt Hogstrom wrote:
>> Lately I've been working with users in debugging various
>> application problems. Some of the problems are merely
>> configuration but others are deeper application / infrastructure
>> problems. Regardless of the type of problem I've never personally
>> been satisfied with the diagnostic information produced by the
>> server (this isn't a Geronimo statement but really AppServers as a
>> whole including WebSphere and WebLogic).
>>
>> Here are some thoughts that I want to pursue:
>>
>> I’ve been working with some customers lately and the work has
>> centered around debugging some of the aspects of their server. In
>> this case it was using Apache Geronimo but the problem really
>> applies to most application servers in general. For the most part
>> there is little diagnostic information available when an
>> application fails. We get the ever popular nested Java Stack
>> trace which is certainly a good indicator of where a failure
>> occurred but is woefully inadequate in many instances of why a
>> failure occurred. This get’s worse in that for the most part
>> people need to recreate the problem with additional tracing and,
>> in the worst case, additional diagnostic code in their
>> application. Wouldn’t it be nice if some of the diagnostic
>> capability that was needed was included in the server itself?
>>
>> Over the next few weeks I’m going to be doing some experimentation
>> on how to improve server diagnostics through the use of Aspects
>> and/or Instrumentation. Since this is experimental we’ll see what
>> the final result will be. Here are some of my initial goals:
>>
>> 1. Improve diagnostics by providing a Diagnostic Report when a
>> Tx fails.
>> 2. Provide better visualization of Java Stack traces so problem
>> areas pop out.
>> 3. Capture wait information
>>
>> For number 1 I’m going to focus on servlets to begin with given
>> that they represent the preponderance of requests made in
>> AppServers today. This information will include information from
>> the request object, the servlet being invoked, invocation time,
>> transaction ID (if it exists), enlisted connections (database and
>> messaging), oh yeah, and the Thread ID of execution. This is a
>> mouthful to begin with anyway.
>>
>> Number 2 is really just applying some template information on a
>> Java Stack trace. I want application classes to standout so
>> developers will be able to quickly see where their application is
>> involved. Infrastructure pieces like the server, Hibernate,
>> TopLink, etc. would also be highlighted in a different color and
>> style and plain old java classes would be a boring style as their
>> are merely pawns in the transactional game.
>>
>> Finally, wouldn’t it be nice to know how long a thread has been
>> waiting and for what reason? Is it waiting on a request from
>> another server or perhaps there is a database locking problem.
>> Did a WebService go awry? Basically, I want to know what in the
>> heck a thread is waiting on.
>>
>> Please chime in with your thoughts. I spect that Aspects or
>> Instrumentation may be the only way to go for some of this as many
>> of the components we include won't have this capability in them.
>
>
Re: [DISCUSS] Improving diagnostics
Posted by Rick McGuire <ri...@gmail.com>.
One thing that might be a useful first start would be something similar
to the "Hungry Exception" cleanup effort that was done with openejb3.
Basically, this just ensured that any place where an exception gets
thrown because of another caught exception that the original exception
was maintained as a cause. I've spend a lot of time grumbling about
situations where the original exception information was thrown away,
sometimes through multiple levels of failure.
Rick
Matt Hogstrom wrote:
> Lately I've been working with users in debugging various application
> problems. Some of the problems are merely configuration but others
> are deeper application / infrastructure problems. Regardless of the
> type of problem I've never personally been satisfied with the
> diagnostic information produced by the server (this isn't a Geronimo
> statement but really AppServers as a whole including WebSphere and
> WebLogic).
>
> Here are some thoughts that I want to pursue:
>
> I’ve been working with some customers lately and the work has centered
> around debugging some of the aspects of their server. In this case it
> was using Apache Geronimo but the problem really applies to most
> application servers in general. For the most part there is little
> diagnostic information available when an application fails. We get
> the ever popular nested Java Stack trace which is certainly a good
> indicator of where a failure occurred but is woefully inadequate in
> many instances of why a failure occurred. This get’s worse in that
> for the most part people need to recreate the problem with additional
> tracing and, in the worst case, additional diagnostic code in their
> application. Wouldn’t it be nice if some of the diagnostic capability
> that was needed was included in the server itself?
>
> Over the next few weeks I’m going to be doing some experimentation on
> how to improve server diagnostics through the use of Aspects and/or
> Instrumentation. Since this is experimental we’ll see what the final
> result will be. Here are some of my initial goals:
>
> 1. Improve diagnostics by providing a Diagnostic Report when a Tx
> fails.
> 2. Provide better visualization of Java Stack traces so problem
> areas pop out.
> 3. Capture wait information
>
> For number 1 I’m going to focus on servlets to begin with given that
> they represent the preponderance of requests made in AppServers
> today. This information will include information from the request
> object, the servlet being invoked, invocation time, transaction ID (if
> it exists), enlisted connections (database and messaging), oh yeah,
> and the Thread ID of execution. This is a mouthful to begin with anyway.
>
> Number 2 is really just applying some template information on a Java
> Stack trace. I want application classes to standout so developers
> will be able to quickly see where their application is involved.
> Infrastructure pieces like the server, Hibernate, TopLink, etc. would
> also be highlighted in a different color and style and plain old java
> classes would be a boring style as their are merely pawns in the
> transactional game.
>
> Finally, wouldn’t it be nice to know how long a thread has been
> waiting and for what reason? Is it waiting on a request from another
> server or perhaps there is a database locking problem. Did a
> WebService go awry? Basically, I want to know what in the heck a
> thread is waiting on.
>
> Please chime in with your thoughts. I spect that Aspects or
> Instrumentation may be the only way to go for some of this as many of
> the components we include won't have this capability in them.