You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Shanti Suresh <sh...@umich.edu> on 2014/04/15 16:51:31 UTC
Performance - Java Profiler, JVM instrmentation
Greetings,
Chris' presentation on monitoring Tomcat is really nice work. I found that
quite useful.
Taking it one step further, could I request some recommendations on how we
might profile Java code running inside Tomcat? Often, I am stuck with
finding out why an application is slow. It could be a consistent,
progressive or a sudden problem. These applications do not expose metrics
via MBeans. Say, for e.g., a vendor application which has been heavily
customized in-house.
Some metrics that I find useful during these times are things like
concurrent invocations, stall counts on components, call-stack,
response-rate etc.
Java Melody has a nice built-in dashboard of metrics. Co-relating metrics
like that is powerful and helps isolate relatively easy problems. I find
that the metrics skim the surface of more involved problems.
In Tomcat, is there a way to go deeper into the performance of the code for
root-cause analysis and isolate a section of the code or a flow in the code
for troubleshooting? How would one go about getting to that place? Let's
say, there is no budget for purchasing tools in that space. I find Chris'
example on writing filters to map to URL patterns for response-time metrics
relevant. I would also like stall counts, concurrent invocations etc.
Greatly appreciate your thoughts and opinions.
Thanks,
-Shanti
Re: Performance - Java Profiler, JVM instrmentation
Posted by Frederik Nosi <fr...@postecom.it>.
Hi Shanti,
On 04/15/2014 09:56 PM, Shanti Suresh wrote:
[...]
>>> I find Chris' example on writing filters to map to URL patterns for
>>> response-time metrics relevant. I would also like stall counts,
>>> concurrent invocations etc.
>> What is a stall-count? How would you record "concurrent invocations",
>> etc.?
>>
> So here is my understanding of these metrics:
>
> So if a request for a servlet or JSP exceeds a given time interval, that
> would be a stall. The interval may depend upon the application. In some
> cases, 10 seconds would be considered a stall, some cases, 30 seconds would
> be a stall.
This can be done enabling the access log and adding a %D on the log
format string, here's
what i add to server.xml in tomcat 6:
<!-- -->
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="access." suffix=".log" resolveHosts="false"
pattern='%h %u %t "%r" %s %b %I %D'
buffered="false" />
then you get another log file, in this case access.DATE.log where the
last entry is the time in milliseconds
it took to complete the request.
Than just do a:
cat access.DATE.log | awk '{ if ($NF > DURATION) { print $0 } }'
Hope you got the idea
>
> Similarly, how many times a servlet is invoked in a given time period would
> count as concurrent invocations. Intervals used for the reckoning here may
> be shorter - like 5 seconds - to make it more meaningful for concurrency
> values.
You can use the access log for this too
[..]
Frederik
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
Posted by Shanti Suresh <sh...@umich.edu>.
On Tue, Apr 15, 2014 at 3:59 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
>
>
> Can you afford to wait for the request to complete (late), or do you
> need to know immediately when a request is taking too long?
>
My reasoning for measuring the long-running requests at a given period of
time is so that you can co-relate it to other metrics at that given time.
In my homegrown dashboard, I would graph the values returned alongside CPU,
thread-use, Heap-size, connection-pool size etc. And then try to locate a
signature of the problem. So just requests that have been inflight for the
last "n" seconds at a given period of time would help. Or again, late
requests, like you mention might work too. So at a given time, which
requests completed late.
Some of the tools out there have this troubleshooting down to a science and
it makes life so easy for all parties!
> In either case, this type of thing can be done easily with a filter.
> Whether you use JMX or not to report the condition is up to you.
>
> > Similarly, how many times a servlet is invoked in a given time
> > period would count as concurrent invocations. Intervals used for
> > the reckoning here may be shorter - like 5 seconds - to make it
> > more meaningful for concurrency values.
>
> Again, Filters are your friends. Feel free to publish the information
> via JMX as well. My presentation contains all the information you need
> for the JMX stuff. Everything else is pure Java.
>
>
> Yes, I plan to explore that more.
Thanks again,
-Shanti
Re: Performance - Java Profiler, JVM instrmentation
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Shanti,
On 4/15/14, 3:56 PM, Shanti Suresh wrote:
> So here is my understanding of these metrics:
>
> So if a request for a servlet or JSP exceeds a given time interval,
> that would be a stall. The interval may depend upon the
> application. In some cases, 10 seconds would be considered a
> stall, some cases, 30 seconds would be a stall.
Can you afford to wait for the request to complete (late), or do you
need to know immediately when a request is taking too long?
In either case, this type of thing can be done easily with a filter.
Whether you use JMX or not to report the condition is up to you.
> Similarly, how many times a servlet is invoked in a given time
> period would count as concurrent invocations. Intervals used for
> the reckoning here may be shorter - like 5 seconds - to make it
> more meaningful for concurrency values.
Again, Filters are your friends. Feel free to publish the information
via JMX as well. My presentation contains all the information you need
for the JMX stuff. Everything else is pure Java.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTTY+rAAoJEBzwKT+lPKRYApcP/ROvCHanDW2RgYEneUPoG2u9
g/u2IKq8Noe8RmeQpVuU+k5i3tUo4hEigYsnmBiYHVq+CxY31hkMiPtuwMgHIrPd
rHGfCy3Mgpsc3yLI2yuvYqCSs3pqR3+PYO1CiaxmgT52W8JWGoQjOOOdlk0BmseZ
kF/vVtkT3qsM+c5YvghsqCnq9dyZK8YQXfzEigFdmLDzA42jAJ9gPhmoU4ZtfIXZ
MKsn2xKgsz4XWaLsgLL2U/JX9Q77Sn/vXCcWktvP6wPZJDGFwf2jKKTuOF182NcT
fevPjIysxi/R+zDUh+Dw7NkKty6HMx0x8DZHhKsy61YeK5fU+PgY8eyinGz5uTa2
+p2IgX2fUG5MiGe5oSO4g2oqLwNCqFsRbsvY5N3N4SnxuXaeQIsaIQKqYzgxBQyE
d6BkPCGsObnpS59pqiQ2rF1cYNyu4z6pILGfS6ctZn43SSJ60HmhBZvE/qRtsezQ
ycC2Wu8YzF8kJwCJyJJyPpg59m+hhlWYQ/Uk1BgTEhYpAYoAeOgrog+gdFXYt1RZ
buI/t5vX6i38RjSe/L1zgDpx/x1x5kn4H/zhUfKM//4HaInfN2KG4cVejAhGtr5X
kunZCIwh6rVlDIV167aIL6qDXinCpIIWJN2FhKO7khtCQrrbIIp7NYwWG0XfHYTn
X4Fq69OJZRfKf7lDcSHv
=uclh
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
Posted by Shanti Suresh <sh...@umich.edu>.
Hi Chris,
On Tue, Apr 15, 2014 at 1:58 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> You need to use a Profiler for that. There are a number of fine
> Profilers available for Java. I use YourKit because they give free
> licenses to ASF committers.
>
I'll look into a Java Profiler.
>
>
> Tomcat provides no such tools. You can use jconsole for a bit of
> profiling, but I've never used it heavily.
>
I see.
>
> Honestly, if there is no budget for such tools, then there is no
> budget for improving the performance of your application. You'll have
> to convince someone with a higher level of clearance that customer
> needs are worth spending money.
>
I am trying.
>
> > I find Chris' example on writing filters to map to URL patterns for
> > response-time metrics relevant. I would also like stall counts,
> > concurrent invocations etc.
>
> What is a stall-count? How would you record "concurrent invocations",
> etc.?
>
So here is my understanding of these metrics:
So if a request for a servlet or JSP exceeds a given time interval, that
would be a stall. The interval may depend upon the application. In some
cases, 10 seconds would be considered a stall, some cases, 30 seconds would
be a stall.
Similarly, how many times a servlet is invoked in a given time period would
count as concurrent invocations. Intervals used for the reckoning here may
be shorter - like 5 seconds - to make it more meaningful for concurrency
values.
Hi Leon,
On Tue, Apr 15, 2014 at 3:45 PM, Leon Rosenberg <ro...@gmail.com>wrote:
>
>
> last time I tried to use a profile on a production site it killed it within
> a second. Usually the performance overhead of a profiler is so huge, that
> you have no chance to run it in production.
> And real problems do not occur in test labs ;-)
>
> Leon
>
I need to revisit New Relic. Many setups use these tools against QA and
don't pass new code until load tests exercising the new code do not trip
any thresholds. Which is a good workflow, although testing is controlled.
I've only used them against production.
Thanks,
-Shanti
Re: Performance - Java Profiler, JVM instrmentation
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Leon,
On 4/15/14, 3:45 PM, Leon Rosenberg wrote:
> On Tue, Apr 15, 2014 at 7:58 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> Shanti,
>>
>> On 4/15/14, 10:51 AM, Shanti Suresh wrote:
>>> Taking it one step further, could I request some
>>> recommendations on how we might profile Java code running
>>> inside Tomcat? Often, I am stuck with finding out why an
>>> application is slow. It could be a consistent, progressive or
>>> a sudden problem. These applications do not expose metrics via
>>> MBeans. Say, for e.g., a vendor application which has been
>>> heavily customized in-house.
>>
>> You need to use a Profiler for that. There are a number of fine
>> Profilers available for Java. I use YourKit because they give
>> free licenses to ASF committers.
>>
>> Hello Chris, et al,
>
> last time I tried to use a profile on a production site it killed
> it within a second.
+1
Profiling in production is a terrible idea.
> Usually the performance overhead of a profiler is so huge, that you
> have no chance to run it in production. And real problems do not
> occur in test labs ;-)
Performance problems can usually be reproduced in labs.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTTY8pAAoJEBzwKT+lPKRYezUQALIQ8Vy124z/ZZZum3IqpEhy
F5nk5rdDZLBtQF7wT8WH68fo254MNSvCPqfOLKfwDKGuMEl+RTxFg58k96Ii7e+S
xC6wQwInJefdEtB1jz2E4hic8artLpAfjDPlXCT8DRHdLEWUF3qQ43H994/I3Ybs
go41WFtP2+8aZaZnGQI664TGaERlrOJRu3AfMjU2VErG3kCJeeR/BFXhj7tZu/1Y
8cXt2CouFmKFYl5KNisI/3gfJ8X3hEGowZkGyswzSh+DkWIgxQMM7grnntHoxFtP
iHlzQPdOB+gXmt2e2SISDc7HZNITRiBRvas8Q/+/xMiNJTbrRuKdRST6YKp3+AIt
KuobyjJixJ0GuwjqW7P5/v+rP/PgF+eAT79FKtvKELUuRophh6e1ifO7osNkAkyp
qCi+HoiThS9DfjJHdQnK9+uGMxRCirTvVIoNqHrJP4WJkxFeVRJQN1OKWaw+n/Xn
SPAehFUZiUgS07NaR79lPvkHl8kXOAvrEEREwW6copv5c4BTQ/WlNYPG+2oRWmeZ
YqTBS4bbZclV69Pg6cSSbfZGF/kmYq5zBKDOGIUHPF2zX4IoPmraU8IOLElIiS7o
0a5P64SLAsvOs4KnRJ4nFZSN80MwV6rODI2376+LlcdLjlQCmxoRiyNJMTOEqMUy
dDn0cVrr7PzF0fpWoQN9
=F8p+
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
Posted by Leon Rosenberg <ro...@gmail.com>.
On Tue, Apr 15, 2014 at 7:58 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Shanti,
>
> On 4/15/14, 10:51 AM, Shanti Suresh wrote:
> > Taking it one step further, could I request some recommendations on
> > how we might profile Java code running inside Tomcat? Often, I am
> > stuck with finding out why an application is slow. It could be a
> > consistent, progressive or a sudden problem. These applications
> > do not expose metrics via MBeans. Say, for e.g., a vendor
> > application which has been heavily customized in-house.
>
> You need to use a Profiler for that. There are a number of fine
> Profilers available for Java. I use YourKit because they give free
> licenses to ASF committers.
>
> Hello Chris, et al,
last time I tried to use a profile on a production site it killed it within
a second. Usually the performance overhead of a profiler is so huge, that
you have no chance to run it in production.
And real problems do not occur in test labs ;-)
Leon
Re: Performance - Java Profiler, JVM instrmentation
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Shanti,
On 4/15/14, 10:51 AM, Shanti Suresh wrote:
> Taking it one step further, could I request some recommendations on
> how we might profile Java code running inside Tomcat? Often, I am
> stuck with finding out why an application is slow. It could be a
> consistent, progressive or a sudden problem. These applications
> do not expose metrics via MBeans. Say, for e.g., a vendor
> application which has been heavily customized in-house.
You need to use a Profiler for that. There are a number of fine
Profilers available for Java. I use YourKit because they give free
licenses to ASF committers.
> In Tomcat, is there a way to go deeper into the performance of the
> code for root-cause analysis and isolate a section of the code or a
> flow in the code for troubleshooting? How would one go about
> getting to that place? Let's say, there is no budget for
> purchasing tools in that space.
Tomcat provides no such tools. You can use jconsole for a bit of
profiling, but I've never used it heavily.
Honestly, if there is no budget for such tools, then there is no
budget for improving the performance of your application. You'll have
to convince someone with a higher level of clearance that customer
needs are worth spending money.
> I find Chris' example on writing filters to map to URL patterns for
> response-time metrics relevant. I would also like stall counts,
> concurrent invocations etc.
What is a stall-count? How would you record "concurrent invocations",
etc.?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTTXM7AAoJEBzwKT+lPKRY/g8P/3hdbqYf5I2s0LaxiHexJ27I
tTouN/Gk0+kmOapdoPUxXECC8q/LF51I7aCF5L2D/XY5Jgu5Vx5grwvoZWVdAS5N
OB54gLvOq1toVSGTI7C7POqg8S7gqdj1Ec5BCrxiI0HGmSWNYcDQoFNWXL+srE6L
xes6o1Kfhlvs/cjmhWagooaGvaNY1Wu7DyhJowaTmWzDecvEmIS+A3ysuSoB+g4B
Myc61ZR4Imul6+0iRm59dWjE5ID9cqK9jr5NzFUIcy2zJaCIkNV1SKnliWkZBgp/
b/B7pUWfTnjUiX4lEDjH9WYL6cXO/fkwL70yue1iki/u+MqOJn7c+Qyo2VzRtn1W
iYHpm5BKngNER8r/4AhzV7ZvqGlmqOoLHqmh88Seiq8AAcloLdDkXWhR/GQzHkuk
+ESCRv5qUmmDwOdk1wcuMjGOk10+tN5zc2ZfrJlchYxVKvPaxHFcFVSczlcb+Bok
heetKDK2PZlYlzQ0iACwBHLzmaU/Jycmv3+zVOucBmyeZZBUX6GpAjG2lsh0QPlG
DhIPj1PHuC6OYsIGbVeLFirFRo/hS+XwACaY/qBVyyGd9k6NRjtpQ9RMQvb2dUGu
CPh52SBzgyOGGu9dZ1EaYSC7Vl21DySDmWr8eUMxIjEYinr0rc5Mv1VqOQEWaWy9
rixRAxhweuazzCqGnat7
=0i+1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
Posted by Leon Rosenberg <ro...@gmail.com>.
On Tue, Apr 15, 2014 at 4:51 PM, Shanti Suresh <sh...@umich.edu> wrote:
> Greetings,
>
Hello Shanti,
>
> Chris' presentation on monitoring Tomcat is really nice work. I found that
> quite useful.
>
> Taking it one step further, could I request some recommendations on how we
> might profile Java code running inside Tomcat? Often, I am stuck with
> finding out why an application is slow. It could be a consistent,
> progressive or a sudden problem. These applications do not expose metrics
> via MBeans. Say, for e.g., a vendor application which has been heavily
> customized in-house.
>
> Some metrics that I find useful during these times are things like
> concurrent invocations, stall counts on components, call-stack,
> response-rate etc.
> Java Melody has a nice built-in dashboard of metrics. Co-relating metrics
> like that is powerful and helps isolate relatively easy problems. I find
> that the metrics skim the surface of more involved problems.
>
> In Tomcat, is there a way to go deeper into the performance of the code for
> root-cause analysis and isolate a section of the code or a flow in the code
> for troubleshooting? How would one go about getting to that place? Let's
> say, there is no budget for purchasing tools in that space. I find Chris'
> example on writing filters to map to URL patterns for response-time metrics
> relevant. I would also like stall counts, concurrent invocations etc.
>
There are tools that are doing exactly that for about 7 years out now.
You can go to http://newrelic.com and get it for as much as 150 USD per
server.
Or you can get all the same for free from http://www.moskito.org. And more.
regards
Leon
>
> Greatly appreciate your thoughts and opinions.
>
> Thanks,
>
> -Shanti
>