Ignite internal events tracing

[Alexey Goncharuk <al...@gmail.com>]

Igniters,

I've recently stumbled across a situation when occasionally Ignite
transactions commit may take up to several seconds while in general most of
the transactions completed in a period of milliseconds.

After a few attempts to analyze this situation with logs, I realized that
this is a no-go and I need a finer instrument for this. The idea is to
introduce several trace points along the way of an Ignite operation and
collect timings when an operation passes each of the trace points. When
enabled, this information should be available upon the operation completion.

I've implemented a prototype of this for TX commit operation, the
implementation is available in ignite-5797 branch.

I was wondering if something of this kind may be useful as a part of Ignite
product and available to users. If so, I would like to discuss the public
API for this so the feature can be finalized.

Thanks,
AG

Re: Ignite internal events tracing

[Yakov Zhdanov <yz...@apache.org>]

Val, it is used by Ignite internals to report the hangs and output
diagnostic information.

--Yakov

Re: Ignite internal events tracing

[Valentin Kulichenko <va...@gmail.com>]

Yakov,

How IgniteDiagnosticAware can be used? Is there any information?

-Val

On Mon, Jul 24, 2017 at 3:24 AM, Yakov Zhdanov <yz...@apache.org> wrote:

> Alex, I like the idea very much, but I think we need to rethink the
> implementation approach to make it more generic. Passing parameter to each
> invocation seems dirty to me.
>
> Val,  we already have this. Please
> see org.apache.ignite.internal.IgniteDiagnosticAware
>
> Dmitry, what you suggest will be pretty hard to implement. I would better
> improve self-diagnostic system to extend list of metrics Ignite monitors.
>
> --Yakov
>

Re: Ignite internal events tracing

[Dmitriy Setrakyan <ds...@apache.org>]

On Mon, Jul 24, 2017 at 5:24 AM, Yakov Zhdanov <yz...@apache.org> wrote:

> Alex, I like the idea very much, but I think we need to rethink the
> implementation approach to make it more generic. Passing parameter to each
> invocation seems dirty to me.
>
> Val,  we already have this. Please
> see org.apache.ignite.internal.IgniteDiagnosticAware
>
> Dmitry, what you suggest will be pretty hard to implement. I would better
> improve self-diagnostic system to extend list of metrics Ignite monitors.
>

Yakov, I was thinking something along the lines of
"enableTransactionTracing(true)" or "enableSqlTracing" and the JMX or API
level. Why should it be hard?

Re: Ignite internal events tracing

[Yakov Zhdanov <yz...@apache.org>]

Alex, I like the idea very much, but I think we need to rethink the
implementation approach to make it more generic. Passing parameter to each
invocation seems dirty to me.

Val,  we already have this. Please
see org.apache.ignite.internal.IgniteDiagnosticAware

Dmitry, what you suggest will be pretty hard to implement. I would better
improve self-diagnostic system to extend list of metrics Ignite monitors.

--Yakov

Re: Ignite internal events tracing

[Dmitriy Setrakyan <ds...@apache.org>]

I think this is a great idea, and should be implemented not only for
transactions, but also for SQL queries. Users should be able to turn this
tracing on and off through JMX and API. I am assuming that this information
will be dumped into the log as well, right?

D.

On Fri, Jul 21, 2017 at 12:04 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:

> Alex,
>
> That's a great idea. I would also add an option to dump information on
> demand, for case when operation hanged and can't complete.
>
> -Val
>
> On Fri, Jul 21, 2017 at 6:15 AM, Alexey Goncharuk <
> alexey.goncharuk@gmail.com> wrote:
>
> > Igniters,
> >
> > I've recently stumbled across a situation when occasionally Ignite
> > transactions commit may take up to several seconds while in general most
> of
> > the transactions completed in a period of milliseconds.
> >
> > After a few attempts to analyze this situation with logs, I realized that
> > this is a no-go and I need a finer instrument for this. The idea is to
> > introduce several trace points along the way of an Ignite operation and
> > collect timings when an operation passes each of the trace points. When
> > enabled, this information should be available upon the operation
> > completion.
> >
> > I've implemented a prototype of this for TX commit operation, the
> > implementation is available in ignite-5797 branch.
> >
> > I was wondering if something of this kind may be useful as a part of
> Ignite
> > product and available to users. If so, I would like to discuss the public
> > API for this so the feature can be finalized.
> >
> > Thanks,
> > AG
> >
>

Re: Ignite internal events tracing

[Valentin Kulichenko <va...@gmail.com>]

Alex,

That's a great idea. I would also add an option to dump information on
demand, for case when operation hanged and can't complete.

-Val

On Fri, Jul 21, 2017 at 6:15 AM, Alexey Goncharuk <
alexey.goncharuk@gmail.com> wrote:

> Igniters,
>
> I've recently stumbled across a situation when occasionally Ignite
> transactions commit may take up to several seconds while in general most of
> the transactions completed in a period of milliseconds.
>
> After a few attempts to analyze this situation with logs, I realized that
> this is a no-go and I need a finer instrument for this. The idea is to
> introduce several trace points along the way of an Ignite operation and
> collect timings when an operation passes each of the trace points. When
> enabled, this information should be available upon the operation
> completion.
>
> I've implemented a prototype of this for TX commit operation, the
> implementation is available in ignite-5797 branch.
>
> I was wondering if something of this kind may be useful as a part of Ignite
> product and available to users. If so, I would like to discuss the public
> API for this so the feature can be finalized.
>
> Thanks,
> AG
>