Best practices for profiling

[Aryeh Friedman <ar...@gmail.com>]

What are the best practices for (open source) profiling a JSP (no taglibs
or any EE outside of JSP and JDBC) webapp running on a remote machine all
the way down the method level (our main user is complaining about
performance and we have done all the easy fixes that can be done without
profiling).   I have played with VisualVM and while it was helpful in
finding the grosser high level stuff it was not the best for pinpointing
specific bottlenecks in the actual code.

-- 
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org

Re: Best practices for profiling

[Martin Grigorov <mg...@apache.org>]

Hi,

On Sun, Aug 2, 2020 at 1:09 AM calder <ca...@gmail.com> wrote:

> On Sat, Aug 1, 2020, 15:16 Aryeh Friedman <ar...@gmail.com>
> wrote:
>
> > On Sat, Aug 1, 2020 at 3:28 PM calder <ca...@gmail.com> wrote:
> >
> > > On Sat, Aug 1, 2020 at 12:53 PM Aryeh Friedman <
> aryeh.friedman@gmail.com
> > >
> > > wrote:
> > > >
> > > > What are the best practices for (open source) profiling a JSP (no
> > taglibs
> > >
> > > There's no "one rule" for this
> > >
> > > > or any EE outside of JSP and JDBC) webapp running on a remote machine
> > all
> > > > the way down the method level (our main user is complaining about
> > > > performance and we have done all the easy fixes that can be done
> > without
> > > > profiling).   I have played with VisualVM and while it was helpful in
> > > > finding the grosser high level stuff it was not the best for
> > pinpointing
> > > > specific bottlenecks in the actual code.
> > >
> >
> >
> > > [1] We prefer to not profile / debug in a client's environment,
> > > because there may be "too much going on elsewhere".
> > >
> >
> > Ours is a custom web app (remote cardiac monitoring/record keeping) that
> > has only one production server and thus might make the choices narrower.
> > The primary problem is it runs just fine on our development machines
> (even
> > under stress tests) because we have more efficent setup then what their
> > hosting company is willing to provide (we already use VM's and have them
> > backed by local SSD's w/ traditional file systems but the hosting company
> > insists on doing networked object storage [which is impossible to model
> > outside of that environment]).   When not loaded outside of peak business
> > hours it also runs just fine on the hosting company.   Thus it comes down
> > to a problem only when it is under a production load and the client is
> not
> > willing to pay for a duplicate setup at the hosting firm (nor would it
> let
> > us see it under load).   Our main goal in profiling is to identify which
> > classes/methods that are the bottlenecks.   We are fairly sure it is
> > backend I/O driven and just need to see which of the I/O calls are the
> big
> > offenders so we can work out ways of avoiding making them.
> >
> > As to debugging except for confirming the bug in production and/or using
> > its logs to pinpoint the bug we do not do debugging in production.
> >
>
> You'll probably want to look at an APM type, such as AppDynamics
>

Before doing this I'd suggest you to try something simpler: ask them to
provide you with thread dumps while the system is under load.
I.e. whenever they feel the system is slow someone who has access to Tomcat
should take several thread dumps (e.g. 3-5 seconds between each dump). They
could use jstack, jcmd, kill -3, VisualVM ...
From the stack traces you could figure out which methods are busiest and
what kind of network calls are being done

Re: Best practices for profiling

[calder <ca...@gmail.com>]

On Sat, Aug 1, 2020, 15:16 Aryeh Friedman <ar...@gmail.com> wrote:

> On Sat, Aug 1, 2020 at 3:28 PM calder <ca...@gmail.com> wrote:
>
> > On Sat, Aug 1, 2020 at 12:53 PM Aryeh Friedman <aryeh.friedman@gmail.com
> >
> > wrote:
> > >
> > > What are the best practices for (open source) profiling a JSP (no
> taglibs
> >
> > There's no "one rule" for this
> >
> > > or any EE outside of JSP and JDBC) webapp running on a remote machine
> all
> > > the way down the method level (our main user is complaining about
> > > performance and we have done all the easy fixes that can be done
> without
> > > profiling).   I have played with VisualVM and while it was helpful in
> > > finding the grosser high level stuff it was not the best for
> pinpointing
> > > specific bottlenecks in the actual code.
> >
>
>
> > [1] We prefer to not profile / debug in a client's environment,
> > because there may be "too much going on elsewhere".
> >
>
> Ours is a custom web app (remote cardiac monitoring/record keeping) that
> has only one production server and thus might make the choices narrower.
> The primary problem is it runs just fine on our development machines (even
> under stress tests) because we have more efficent setup then what their
> hosting company is willing to provide (we already use VM's and have them
> backed by local SSD's w/ traditional file systems but the hosting company
> insists on doing networked object storage [which is impossible to model
> outside of that environment]).   When not loaded outside of peak business
> hours it also runs just fine on the hosting company.   Thus it comes down
> to a problem only when it is under a production load and the client is not
> willing to pay for a duplicate setup at the hosting firm (nor would it let
> us see it under load).   Our main goal in profiling is to identify which
> classes/methods that are the bottlenecks.   We are fairly sure it is
> backend I/O driven and just need to see which of the I/O calls are the big
> offenders so we can work out ways of avoiding making them.
>
> As to debugging except for confirming the bug in production and/or using
> its logs to pinpoint the bug we do not do debugging in production.
>

You'll probably want to look at an APM type, such as AppDynamics

Re: Best practices for profiling

[Aryeh Friedman <ar...@gmail.com>]

On Sat, Aug 1, 2020 at 3:28 PM calder <ca...@gmail.com> wrote:

> On Sat, Aug 1, 2020 at 12:53 PM Aryeh Friedman <ar...@gmail.com>
> wrote:
> >
> > What are the best practices for (open source) profiling a JSP (no taglibs
>
> There's no "one rule" for this
>
> > or any EE outside of JSP and JDBC) webapp running on a remote machine all
> > the way down the method level (our main user is complaining about
> > performance and we have done all the easy fixes that can be done without
> > profiling).   I have played with VisualVM and while it was helpful in
> > finding the grosser high level stuff it was not the best for pinpointing
> > specific bottlenecks in the actual code.
>


> [1] We prefer to not profile / debug in a client's environment,
> because there may be "too much going on elsewhere".
>

Ours is a custom web app (remote cardiac monitoring/record keeping) that
has only one production server and thus might make the choices narrower.
The primary problem is it runs just fine on our development machines (even
under stress tests) because we have more efficent setup then what their
hosting company is willing to provide (we already use VM's and have them
backed by local SSD's w/ traditional file systems but the hosting company
insists on doing networked object storage [which is impossible to model
outside of that environment]).   When not loaded outside of peak business
hours it also runs just fine on the hosting company.   Thus it comes down
to a problem only when it is under a production load and the client is not
willing to pay for a duplicate setup at the hosting firm (nor would it let
us see it under load).   Our main goal in profiling is to identify which
classes/methods that are the bottlenecks.   We are fairly sure it is
backend I/O driven and just need to see which of the I/O calls are the big
offenders so we can work out ways of avoiding making them.

As to debugging except for confirming the bug in production and/or using
its logs to pinpoint the bug we do not do debugging in production.


> We create a VM and use their OS, then install our software per our
> specs, using the AppServer they have chosen.
>
> If we see better performance in our client-replicated environment (our VM),
> than in theirs, we suggest to them to get with their DevOps team to
> find the "other source" that is contributing to their performance
> issue - that may involve the vendor of said redirector/etc.
>
> ** One thing to keep in mind - when you are testing, test directly to
> the AppServer (for example, browser <-> AppServer) and do NOT use a
> redirector [in front of the AppServer] that the client might be using.
> We have discovered 80% of client performance issues ended up being
> with their redirector).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

-- 
Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org

Re: Best practices for profiling

[calder <ca...@gmail.com>]

On Sat, Aug 1, 2020 at 12:53 PM Aryeh Friedman <ar...@gmail.com> wrote:
>
> What are the best practices for (open source) profiling a JSP (no taglibs

There's no "one rule" for this

> or any EE outside of JSP and JDBC) webapp running on a remote machine all
> the way down the method level (our main user is complaining about
> performance and we have done all the easy fixes that can be done without
> profiling).   I have played with VisualVM and while it was helpful in
> finding the grosser high level stuff it was not the best for pinpointing
> specific bottlenecks in the actual code.

You can't rely on **one single** profiler - we use multiple profilers,
because they each have their strengths and weaknesses.
We write software that is both in our Hosted environment and
installable in the client environment. We support WebLogic, WebSphere,
and Tomcat. There are occasions where we need to profile / debug in
our Hosted environment ... or mimicking a client's environment[1].

Basically, we follow the same pattern:
a) Mission Control and VisualVM and Yourkit - for broad strokes
b) Stackify Prefix (or other Java Agent profiler like XRebel, or maybe
an Aspect type)
c) AppDynamics - APM type

We profile / debug in the steps outlined above - we use (a) to see if
we can nail down the issue - if not, we move to (b) and (c).
Don't take this list of products as gospel - there are others, and
maybe you already have a license for a comparable toolset.

The key is, "don't rely on just one toolset".

[1] We prefer to not profile / debug in a client's environment,
because there may be "too much going on elsewhere".

We create a VM and use their OS, then install our software per our
specs, using the AppServer they have chosen.

If we see better performance in our client-replicated environment (our VM),
than in theirs, we suggest to them to get with their DevOps team to
find the "other source" that is contributing to their performance
issue - that may involve the vendor of said redirector/etc.

** One thing to keep in mind - when you are testing, test directly to
the AppServer (for example, browser <-> AppServer) and do NOT use a
redirector [in front of the AppServer] that the client might be using.
We have discovered 80% of client performance issues ended up being
with their redirector).

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org