You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Vladimir Ozerov <vo...@gridgain.com> on 2016/01/29 21:31:02 UTC
Adding tracing component to Ignite
Folks,
Quite often we face some concurrency or performance issues. And what is
worse, sometimes they spread across several nodes making them hard to debug.
We do various things to debug such problems. We add System.outs, measure
latencies between algorithm steps, etc.. And once the problem is resolved,
we throw away all debugging code.
What if add a kind of tracing component to Ignite, so that convenient
debugging infrastructure wiil always be ready for use?
E.g., assume that we have performance issue in cache operation. Tracing
framework could be used as follows:
/** Register tracer. */
Cache.onStart() {
...
ctx.trace().createProfile("cache_problem");
}
/** Put trace inside problematic code. */
Cache.get(...) {
TraceEvent traceEvt = ctx.trace().start("cache_problem");
...
traceEvt.checkpoint("LOCKED", [optional args and identifiers]);
...
traceEvt.checkpoint("COMMITTED", ...);
}
/** In the end we dump problematic events. */
Cache.onStop() {
...
ctx.profile("cache_problem").dump(System.out, [FILTER PREDICATE]);
}
Result:
23:20:01.234 [LOCKED=20ms, COMMITTED=152ms, FINISHED=153ms]
23:20:03.211 [LOCKED= 1ms, COMMITTED= 1ms, FINISHED= 2ms]
I believe lots of you already did something like this during debug. The
idea is to have some reusable component, so that we do not need to reinvent
the wheel over and over again.
Thoughts?
Vladimir.
Re: Adding tracing component to Ignite
Posted by Sergi Vladykin <se...@gmail.com>.
+1
I think it is a great idea, still it must be carefully designed to avoid
additional performance penalty.
BTW, for performance measurements there is already a good component:
https://github.com/HdrHistogram/HdrHistogram
Sergi
2016-01-30 3:01 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> How do you disable it? I mean, how do you make sure that it does not affect
> performance when it is turned off?
>
> D.
>
> On Fri, Jan 29, 2016 at 12:31 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Folks,
> >
> > Quite often we face some concurrency or performance issues. And what is
> > worse, sometimes they spread across several nodes making them hard to
> > debug.
> >
> > We do various things to debug such problems. We add System.outs, measure
> > latencies between algorithm steps, etc.. And once the problem is
> resolved,
> > we throw away all debugging code.
> >
> > What if add a kind of tracing component to Ignite, so that convenient
> > debugging infrastructure wiil always be ready for use?
> >
> > E.g., assume that we have performance issue in cache operation. Tracing
> > framework could be used as follows:
> >
> > /** Register tracer. */
> > Cache.onStart() {
> > ...
> > ctx.trace().createProfile("cache_problem");
> > }
> >
> > /** Put trace inside problematic code. */
> > Cache.get(...) {
> > TraceEvent traceEvt = ctx.trace().start("cache_problem");
> > ...
> > traceEvt.checkpoint("LOCKED", [optional args and identifiers]);
> > ...
> > traceEvt.checkpoint("COMMITTED", ...);
> > }
> >
> > /** In the end we dump problematic events. */
> > Cache.onStop() {
> > ...
> > ctx.profile("cache_problem").dump(System.out, [FILTER PREDICATE]);
> > }
> >
> > Result:
> > 23:20:01.234 [LOCKED=20ms, COMMITTED=152ms, FINISHED=153ms]
> > 23:20:03.211 [LOCKED= 1ms, COMMITTED= 1ms, FINISHED= 2ms]
> >
> > I believe lots of you already did something like this during debug. The
> > idea is to have some reusable component, so that we do not need to
> reinvent
> > the wheel over and over again.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>
Re: Adding tracing component to Ignite
Posted by Dmitriy Setrakyan <ds...@apache.org>.
How do you disable it? I mean, how do you make sure that it does not affect
performance when it is turned off?
D.
On Fri, Jan 29, 2016 at 12:31 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Folks,
>
> Quite often we face some concurrency or performance issues. And what is
> worse, sometimes they spread across several nodes making them hard to
> debug.
>
> We do various things to debug such problems. We add System.outs, measure
> latencies between algorithm steps, etc.. And once the problem is resolved,
> we throw away all debugging code.
>
> What if add a kind of tracing component to Ignite, so that convenient
> debugging infrastructure wiil always be ready for use?
>
> E.g., assume that we have performance issue in cache operation. Tracing
> framework could be used as follows:
>
> /** Register tracer. */
> Cache.onStart() {
> ...
> ctx.trace().createProfile("cache_problem");
> }
>
> /** Put trace inside problematic code. */
> Cache.get(...) {
> TraceEvent traceEvt = ctx.trace().start("cache_problem");
> ...
> traceEvt.checkpoint("LOCKED", [optional args and identifiers]);
> ...
> traceEvt.checkpoint("COMMITTED", ...);
> }
>
> /** In the end we dump problematic events. */
> Cache.onStop() {
> ...
> ctx.profile("cache_problem").dump(System.out, [FILTER PREDICATE]);
> }
>
> Result:
> 23:20:01.234 [LOCKED=20ms, COMMITTED=152ms, FINISHED=153ms]
> 23:20:03.211 [LOCKED= 1ms, COMMITTED= 1ms, FINISHED= 2ms]
>
> I believe lots of you already did something like this during debug. The
> idea is to have some reusable component, so that we do not need to reinvent
> the wheel over and over again.
>
> Thoughts?
>
> Vladimir.
>