You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by Barney Pippin <ja...@uk.bnpparibas.com> on 2019/05/02 07:52:47 UTC

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Thanks for the response Ilya. Did you get a chance to look at this Pavel?
Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Pavel Tupitsyn <pt...@apache.org>.

>  the invoker thread is going to come back eventually
Invoker thread can be anything, it can be busy, waiting or aborted.
Async is just syntactic sugar to solve "callback hell", compiler transforms
it into a state machine with callbacks.

In general, the behavior is the same in the end - callbacks/continuations
are invoked on the future/task completion thread.
Yes, there are mechanisms that are used in .NET UI frameworks to actually
dispatch the callback to the "main" thread, but
that main thread is controlled by the framework and runs a
producer/consumer queue. But that's not possible for console apps and so on.

Back to our problem - yes, using public pool for callbacks seems to be the
most obvious step.

On Fri, Aug 9, 2019 at 7:31 PM Ilya Kasnacheev <il...@gmail.com>
wrote:

> I'm not sure, but I think that in case of async, the invoker thread is
> going to come back eventually?
>
> In case of Java futures with listeners, I think listeners should be invoked
> from public pool.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 9 авг. 2019 г. в 19:16, Pavel Tupitsyn <pt...@apache.org>:
>
> > Well, you can't just take any running thread and make it run your code
> > instead, right?
> >
> > On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > Why can't we dispatch those callbacks in the caller thread?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn <pt...@apache.org>:
> > >
> > > > Иван,
> > > >
> > > > The fix is to dispatch those callbacks (future listeners) to a
> > different
> > > > thread pool, not sure which one though.
> > > > If I would do a .NET-only fix, I would use the default thread pool
> (non
> > > > Ignite-specific), for Java-side there is no such thing as I
> understand.
> > > >
> > > > Yes, let's have a ticket to track the issue.
> > > >
> > > > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vo...@gmail.com>
> > > wrote:
> > > >
> > > > > Ilya, Pavel,
> > > > >
> > > > > Do we a have a proposal how to fix the root cause of the problem?
> > > > > Should we a have a ticket for it?
> > > > >
> > > > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <
> > ilya.kasnacheev@gmail.com
> > > >:
> > > > > >
> > > > > > Hello!
> > > > > >
> > > > > > I think we should definitely stop running futures out of striped
> > > pool,
> > > > > > while holding any cache logs (stripe thread counts as one).
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > > > > >
> > > > > > > Yes, this can be done purely on .NET side, which is an option
> > that
> > > I
> > > > > > > consider.
> > > > > > > However, the root problem is on Java side, and I believe that
> we
> > > > > should fix
> > > > > > > the root problem.
> > > > > > >
> > > > > > > > violate some of Ignite assumptions: that we never run user
> code
> > > > from
> > > > > > > certain thread pools
> > > > > > > We actually do run user code from Ignite thread pools:
> > > > > > >
> > > > > > > cache.getAsync(1).listen(fut ->
> > > > > > >     System.out.println("Get operation completed [value=" +
> > > fut.get()
> > > > +
> > > > > > > ']'));
> > > > > > >
> > > > > > > `println` here is executed on the striped pool. This is stated
> in
> > > the
> > > > > > > docs that I linked above.
> > > > > > >
> > > > > > > Users have to be aware of this and they have to be very careful
> > > with
> > > > > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > > > > usability.
> > > > > > >
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > > > ilya.kasnacheev@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello!
> > > > > > > >
> > > > > > > > + dev@
> > > > > > > >
> > > > > > > > I think the current behavior, where .Net callbacks may be run
> > > from
> > > > > > > striped
> > > > > > > > pool, violate some of Ignite assumptions: that we never run
> > user
> > > > code
> > > > > > > from
> > > > > > > > certain thread pools (like sys-stripe) and that we try to
> limit
> > > > > options
> > > > > > > of
> > > > > > > > running user-supplied code from our internals.
> > > > > > > >
> > > > > > > > I think that future versions of .Net integration should
> remove
> > > the
> > > > > > > ability
> > > > > > > > of async callbacks to be called from non-user threads, even
> if
> > it
> > > > can
> > > > > > > lead
> > > > > > > > to performance degradation in some cases. I suggest removing
> > this
> > > > > mode,
> > > > > > > if
> > > > > > > > possible, while keeping only the safe one, where internal
> > threads
> > > > > are not
> > > > > > > > waiting upon completion of user code.
> > > > > > > >
> > > > > > > > In this case my issue IGNITE-12033 could be used to track
> this
> > > > work.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > --
> > > > > > > > Ilya Kasnacheev
> > > > > > > >
> > > > > > > >
> > > > > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <
> > ptupitsyn@apache.org
> > > >:
> > > > > > > >
> > > > > > > >> Sorry guys, I've completely missed this thread, and the
> topic
> > is
> > > > > very
> > > > > > > >> important.
> > > > > > > >>
> > > > > > > >> First, a simple fix for the given example. Add the following
> > on
> > > > the
> > > > > > > first
> > > > > > > >> line of Main:
> > > > > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > > > > >> ThreadPoolSynchronizationContext());
> > > > > > > >>
> > > > > > > >> And put the ThreadPoolSynchronizationContext class
> somewhere:
> > > > > > > >> class ThreadPoolSynchronizationContext :
> > SynchronizationContext
> > > > > > > >> {
> > > > > > > >>     // No-op.
> > > > > > > >> }
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Now, detailed explanation. The problem exists forever in
> > Ignite
> > > > and
> > > > > is
> > > > > > > >> mentioned in the docs briefly [1].
> > > > > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > > > > >>
> > > > > > > >> Breakdown:
> > > > > > > >> * Ignite (Java side) runs async callbacks (continuations) on
> > > > system
> > > > > > > >> threads, and those threads have limitations (you should not
> > call
> > > > > Ignite
> > > > > > > >> APIs from them in general)
> > > > > > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > > > > > >> * Usually `await ...` call in .NET will continue execution
> on
> > > the
> > > > > > > >> original Thread (simply put, actually it is more complex),
> so
> > > > Ignite
> > > > > > > system
> > > > > > > >> thread issue is avoided
> > > > > > > >> * However, Console applications have no
> > > `SynchronizationContext`,
> > > > > so the
> > > > > > > >> continuation can't be dispatched to original thread, and is
> > > > > executed on
> > > > > > > >> current (Ignite) thread
> > > > > > > >> * Setting custom SynchronizationContext fixes the issue: all
> > > async
> > > > > > > >> continuations will be dispatched to .NET thread pool and
> never
> > > > > executed
> > > > > > > on
> > > > > > > >> Ignite threads
> > > > > > > >>
> > > > > > > >> However, dispatching callbacks to a different thread causes
> > > > > performance
> > > > > > > >> hit, and Ignite favors performance over usability right now.
> > > > > > > >> So it is up to the user to configure desired behavior.
> > > > > > > >>
> > > > > > > >> Let me know if you need more details.
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >>
> > > > > > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > > > > > >> [2]
> > > https://apacheignite-net.readme.io/docs/asynchronous-support
> > > > > > > >>
> > > > > > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > > > > > ilya.kasnacheev@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Hello!
> > > > > > > >>>
> > > > > > > >>> I have filed a ticket about this issue so it won't get
> lost.
> > > > > > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > > > > > >>>
> > > > > > > >>> Regards,
> > > > > > > >>> --
> > > > > > > >>> Ilya Kasnacheev
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > > > > > > james.prince@uk.bnpparibas.com
> > > > > > > >>> >:
> > > > > > > >>>
> > > > > > > >>>> Thanks for the response Ilya. Did you get a chance to look
> > at
> > > > this
> > > > > > > >>>> Pavel?
> > > > > > > >>>> Thanks.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> --
> > > > > > > >>>> Sent from:
> http://apache-ignite-users.70518.x6.nabble.com/
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Ilya Kasnacheev <il...@gmail.com>.

I'm not sure, but I think that in case of async, the invoker thread is
going to come back eventually?

In case of Java futures with listeners, I think listeners should be invoked
from public pool.

Regards,
-- 
Ilya Kasnacheev


пт, 9 авг. 2019 г. в 19:16, Pavel Tupitsyn <pt...@apache.org>:

> Well, you can't just take any running thread and make it run your code
> instead, right?
>
> On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
> > Hello!
> >
> > Why can't we dispatch those callbacks in the caller thread?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn <pt...@apache.org>:
> >
> > > Иван,
> > >
> > > The fix is to dispatch those callbacks (future listeners) to a
> different
> > > thread pool, not sure which one though.
> > > If I would do a .NET-only fix, I would use the default thread pool (non
> > > Ignite-specific), for Java-side there is no such thing as I understand.
> > >
> > > Yes, let's have a ticket to track the issue.
> > >
> > > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vo...@gmail.com>
> > wrote:
> > >
> > > > Ilya, Pavel,
> > > >
> > > > Do we a have a proposal how to fix the root cause of the problem?
> > > > Should we a have a ticket for it?
> > > >
> > > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com
> > >:
> > > > >
> > > > > Hello!
> > > > >
> > > > > I think we should definitely stop running futures out of striped
> > pool,
> > > > > while holding any cache logs (stripe thread counts as one).
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <pt...@apache.org>:
> > > > >
> > > > > > Yes, this can be done purely on .NET side, which is an option
> that
> > I
> > > > > > consider.
> > > > > > However, the root problem is on Java side, and I believe that we
> > > > should fix
> > > > > > the root problem.
> > > > > >
> > > > > > > violate some of Ignite assumptions: that we never run user code
> > > from
> > > > > > certain thread pools
> > > > > > We actually do run user code from Ignite thread pools:
> > > > > >
> > > > > > cache.getAsync(1).listen(fut ->
> > > > > >     System.out.println("Get operation completed [value=" +
> > fut.get()
> > > +
> > > > > > ']'));
> > > > > >
> > > > > > `println` here is executed on the striped pool. This is stated in
> > the
> > > > > > docs that I linked above.
> > > > > >
> > > > > > Users have to be aware of this and they have to be very careful
> > with
> > > > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > > > usability.
> > > > > >
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > > ilya.kasnacheev@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello!
> > > > > > >
> > > > > > > + dev@
> > > > > > >
> > > > > > > I think the current behavior, where .Net callbacks may be run
> > from
> > > > > > striped
> > > > > > > pool, violate some of Ignite assumptions: that we never run
> user
> > > code
> > > > > > from
> > > > > > > certain thread pools (like sys-stripe) and that we try to limit
> > > > options
> > > > > > of
> > > > > > > running user-supplied code from our internals.
> > > > > > >
> > > > > > > I think that future versions of .Net integration should remove
> > the
> > > > > > ability
> > > > > > > of async callbacks to be called from non-user threads, even if
> it
> > > can
> > > > > > lead
> > > > > > > to performance degradation in some cases. I suggest removing
> this
> > > > mode,
> > > > > > if
> > > > > > > possible, while keeping only the safe one, where internal
> threads
> > > > are not
> > > > > > > waiting upon completion of user code.
> > > > > > >
> > > > > > > In this case my issue IGNITE-12033 could be used to track this
> > > work.
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Regards,
> > > > > > > --
> > > > > > > Ilya Kasnacheev
> > > > > > >
> > > > > > >
> > > > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <
> ptupitsyn@apache.org
> > >:
> > > > > > >
> > > > > > >> Sorry guys, I've completely missed this thread, and the topic
> is
> > > > very
> > > > > > >> important.
> > > > > > >>
> > > > > > >> First, a simple fix for the given example. Add the following
> on
> > > the
> > > > > > first
> > > > > > >> line of Main:
> > > > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > > > >> ThreadPoolSynchronizationContext());
> > > > > > >>
> > > > > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > > > > >> class ThreadPoolSynchronizationContext :
> SynchronizationContext
> > > > > > >> {
> > > > > > >>     // No-op.
> > > > > > >> }
> > > > > > >>
> > > > > > >>
> > > > > > >> Now, detailed explanation. The problem exists forever in
> Ignite
> > > and
> > > > is
> > > > > > >> mentioned in the docs briefly [1].
> > > > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > > > >>
> > > > > > >> Breakdown:
> > > > > > >> * Ignite (Java side) runs async callbacks (continuations) on
> > > system
> > > > > > >> threads, and those threads have limitations (you should not
> call
> > > > Ignite
> > > > > > >> APIs from them in general)
> > > > > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > > > > >> * Usually `await ...` call in .NET will continue execution on
> > the
> > > > > > >> original Thread (simply put, actually it is more complex), so
> > > Ignite
> > > > > > system
> > > > > > >> thread issue is avoided
> > > > > > >> * However, Console applications have no
> > `SynchronizationContext`,
> > > > so the
> > > > > > >> continuation can't be dispatched to original thread, and is
> > > > executed on
> > > > > > >> current (Ignite) thread
> > > > > > >> * Setting custom SynchronizationContext fixes the issue: all
> > async
> > > > > > >> continuations will be dispatched to .NET thread pool and never
> > > > executed
> > > > > > on
> > > > > > >> Ignite threads
> > > > > > >>
> > > > > > >> However, dispatching callbacks to a different thread causes
> > > > performance
> > > > > > >> hit, and Ignite favors performance over usability right now.
> > > > > > >> So it is up to the user to configure desired behavior.
> > > > > > >>
> > > > > > >> Let me know if you need more details.
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > > > > >> [2]
> > https://apacheignite-net.readme.io/docs/asynchronous-support
> > > > > > >>
> > > > > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > > > > ilya.kasnacheev@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Hello!
> > > > > > >>>
> > > > > > >>> I have filed a ticket about this issue so it won't get lost.
> > > > > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>> --
> > > > > > >>> Ilya Kasnacheev
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > > > > > james.prince@uk.bnpparibas.com
> > > > > > >>> >:
> > > > > > >>>
> > > > > > >>>> Thanks for the response Ilya. Did you get a chance to look
> at
> > > this
> > > > > > >>>> Pavel?
> > > > > > >>>> Thanks.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> --
> > > > > > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > > > > > >>>>
> > > > > > >>>
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> >
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Pavel Tupitsyn <pt...@apache.org>.

Well, you can't just take any running thread and make it run your code
instead, right?

On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev <il...@gmail.com>
wrote:

> Hello!
>
> Why can't we dispatch those callbacks in the caller thread?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn <pt...@apache.org>:
>
> > Иван,
> >
> > The fix is to dispatch those callbacks (future listeners) to a different
> > thread pool, not sure which one though.
> > If I would do a .NET-only fix, I would use the default thread pool (non
> > Ignite-specific), for Java-side there is no such thing as I understand.
> >
> > Yes, let's have a ticket to track the issue.
> >
> > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vo...@gmail.com>
> wrote:
> >
> > > Ilya, Pavel,
> > >
> > > Do we a have a proposal how to fix the root cause of the problem?
> > > Should we a have a ticket for it?
> > >
> > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <ilya.kasnacheev@gmail.com
> >:
> > > >
> > > > Hello!
> > > >
> > > > I think we should definitely stop running futures out of striped
> pool,
> > > > while holding any cache logs (stripe thread counts as one).
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <pt...@apache.org>:
> > > >
> > > > > Yes, this can be done purely on .NET side, which is an option that
> I
> > > > > consider.
> > > > > However, the root problem is on Java side, and I believe that we
> > > should fix
> > > > > the root problem.
> > > > >
> > > > > > violate some of Ignite assumptions: that we never run user code
> > from
> > > > > certain thread pools
> > > > > We actually do run user code from Ignite thread pools:
> > > > >
> > > > > cache.getAsync(1).listen(fut ->
> > > > >     System.out.println("Get operation completed [value=" +
> fut.get()
> > +
> > > > > ']'));
> > > > >
> > > > > `println` here is executed on the striped pool. This is stated in
> the
> > > > > docs that I linked above.
> > > > >
> > > > > Users have to be aware of this and they have to be very careful
> with
> > > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > > usability.
> > > > >
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > ilya.kasnacheev@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > + dev@
> > > > > >
> > > > > > I think the current behavior, where .Net callbacks may be run
> from
> > > > > striped
> > > > > > pool, violate some of Ignite assumptions: that we never run user
> > code
> > > > > from
> > > > > > certain thread pools (like sys-stripe) and that we try to limit
> > > options
> > > > > of
> > > > > > running user-supplied code from our internals.
> > > > > >
> > > > > > I think that future versions of .Net integration should remove
> the
> > > > > ability
> > > > > > of async callbacks to be called from non-user threads, even if it
> > can
> > > > > lead
> > > > > > to performance degradation in some cases. I suggest removing this
> > > mode,
> > > > > if
> > > > > > possible, while keeping only the safe one, where internal threads
> > > are not
> > > > > > waiting upon completion of user code.
> > > > > >
> > > > > > In this case my issue IGNITE-12033 could be used to track this
> > work.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > > > > >
> > > > > >> Sorry guys, I've completely missed this thread, and the topic is
> > > very
> > > > > >> important.
> > > > > >>
> > > > > >> First, a simple fix for the given example. Add the following on
> > the
> > > > > first
> > > > > >> line of Main:
> > > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > > >> ThreadPoolSynchronizationContext());
> > > > > >>
> > > > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > > > >> {
> > > > > >>     // No-op.
> > > > > >> }
> > > > > >>
> > > > > >>
> > > > > >> Now, detailed explanation. The problem exists forever in Ignite
> > and
> > > is
> > > > > >> mentioned in the docs briefly [1].
> > > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > > >>
> > > > > >> Breakdown:
> > > > > >> * Ignite (Java side) runs async callbacks (continuations) on
> > system
> > > > > >> threads, and those threads have limitations (you should not call
> > > Ignite
> > > > > >> APIs from them in general)
> > > > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > > > >> * Usually `await ...` call in .NET will continue execution on
> the
> > > > > >> original Thread (simply put, actually it is more complex), so
> > Ignite
> > > > > system
> > > > > >> thread issue is avoided
> > > > > >> * However, Console applications have no
> `SynchronizationContext`,
> > > so the
> > > > > >> continuation can't be dispatched to original thread, and is
> > > executed on
> > > > > >> current (Ignite) thread
> > > > > >> * Setting custom SynchronizationContext fixes the issue: all
> async
> > > > > >> continuations will be dispatched to .NET thread pool and never
> > > executed
> > > > > on
> > > > > >> Ignite threads
> > > > > >>
> > > > > >> However, dispatching callbacks to a different thread causes
> > > performance
> > > > > >> hit, and Ignite favors performance over usability right now.
> > > > > >> So it is up to the user to configure desired behavior.
> > > > > >>
> > > > > >> Let me know if you need more details.
> > > > > >>
> > > > > >> Thanks
> > > > > >>
> > > > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > > > >> [2]
> https://apacheignite-net.readme.io/docs/asynchronous-support
> > > > > >>
> > > > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > > > ilya.kasnacheev@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hello!
> > > > > >>>
> > > > > >>> I have filed a ticket about this issue so it won't get lost.
> > > > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> --
> > > > > >>> Ilya Kasnacheev
> > > > > >>>
> > > > > >>>
> > > > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > > > > james.prince@uk.bnpparibas.com
> > > > > >>> >:
> > > > > >>>
> > > > > >>>> Thanks for the response Ilya. Did you get a chance to look at
> > this
> > > > > >>>> Pavel?
> > > > > >>>> Thanks.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> --
> > > > > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > > > > >>>>
> > > > > >>>
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Ilya Kasnacheev <il...@gmail.com>.

Hello!

Why can't we dispatch those callbacks in the caller thread?

Regards,
-- 
Ilya Kasnacheev


пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn <pt...@apache.org>:

> Иван,
>
> The fix is to dispatch those callbacks (future listeners) to a different
> thread pool, not sure which one though.
> If I would do a .NET-only fix, I would use the default thread pool (non
> Ignite-specific), for Java-side there is no such thing as I understand.
>
> Yes, let's have a ticket to track the issue.
>
> On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vo...@gmail.com> wrote:
>
> > Ilya, Pavel,
> >
> > Do we a have a proposal how to fix the root cause of the problem?
> > Should we a have a ticket for it?
> >
> > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <il...@gmail.com>:
> > >
> > > Hello!
> > >
> > > I think we should definitely stop running futures out of striped pool,
> > > while holding any cache logs (stripe thread counts as one).
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <pt...@apache.org>:
> > >
> > > > Yes, this can be done purely on .NET side, which is an option that I
> > > > consider.
> > > > However, the root problem is on Java side, and I believe that we
> > should fix
> > > > the root problem.
> > > >
> > > > > violate some of Ignite assumptions: that we never run user code
> from
> > > > certain thread pools
> > > > We actually do run user code from Ignite thread pools:
> > > >
> > > > cache.getAsync(1).listen(fut ->
> > > >     System.out.println("Get operation completed [value=" + fut.get()
> +
> > > > ']'));
> > > >
> > > > `println` here is executed on the striped pool. This is stated in the
> > > > docs that I linked above.
> > > >
> > > > Users have to be aware of this and they have to be very careful with
> > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > usability.
> > > >
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > ilya.kasnacheev@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > + dev@
> > > > >
> > > > > I think the current behavior, where .Net callbacks may be run from
> > > > striped
> > > > > pool, violate some of Ignite assumptions: that we never run user
> code
> > > > from
> > > > > certain thread pools (like sys-stripe) and that we try to limit
> > options
> > > > of
> > > > > running user-supplied code from our internals.
> > > > >
> > > > > I think that future versions of .Net integration should remove the
> > > > ability
> > > > > of async callbacks to be called from non-user threads, even if it
> can
> > > > lead
> > > > > to performance degradation in some cases. I suggest removing this
> > mode,
> > > > if
> > > > > possible, while keeping only the safe one, where internal threads
> > are not
> > > > > waiting upon completion of user code.
> > > > >
> > > > > In this case my issue IGNITE-12033 could be used to track this
> work.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <pt...@apache.org>:
> > > > >
> > > > >> Sorry guys, I've completely missed this thread, and the topic is
> > very
> > > > >> important.
> > > > >>
> > > > >> First, a simple fix for the given example. Add the following on
> the
> > > > first
> > > > >> line of Main:
> > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > >> ThreadPoolSynchronizationContext());
> > > > >>
> > > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > > >> {
> > > > >>     // No-op.
> > > > >> }
> > > > >>
> > > > >>
> > > > >> Now, detailed explanation. The problem exists forever in Ignite
> and
> > is
> > > > >> mentioned in the docs briefly [1].
> > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > >>
> > > > >> Breakdown:
> > > > >> * Ignite (Java side) runs async callbacks (continuations) on
> system
> > > > >> threads, and those threads have limitations (you should not call
> > Ignite
> > > > >> APIs from them in general)
> > > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > > >> * Usually `await ...` call in .NET will continue execution on the
> > > > >> original Thread (simply put, actually it is more complex), so
> Ignite
> > > > system
> > > > >> thread issue is avoided
> > > > >> * However, Console applications have no `SynchronizationContext`,
> > so the
> > > > >> continuation can't be dispatched to original thread, and is
> > executed on
> > > > >> current (Ignite) thread
> > > > >> * Setting custom SynchronizationContext fixes the issue: all async
> > > > >> continuations will be dispatched to .NET thread pool and never
> > executed
> > > > on
> > > > >> Ignite threads
> > > > >>
> > > > >> However, dispatching callbacks to a different thread causes
> > performance
> > > > >> hit, and Ignite favors performance over usability right now.
> > > > >> So it is up to the user to configure desired behavior.
> > > > >>
> > > > >> Let me know if you need more details.
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > > > >>
> > > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > > ilya.kasnacheev@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hello!
> > > > >>>
> > > > >>> I have filed a ticket about this issue so it won't get lost.
> > > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > > >>>
> > > > >>> Regards,
> > > > >>> --
> > > > >>> Ilya Kasnacheev
> > > > >>>
> > > > >>>
> > > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > > > james.prince@uk.bnpparibas.com
> > > > >>> >:
> > > > >>>
> > > > >>>> Thanks for the response Ilya. Did you get a chance to look at
> this
> > > > >>>> Pavel?
> > > > >>>> Thanks.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > > > >>>>
> > > > >>>
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Pavel Tupitsyn <pt...@apache.org>.

Иван,

The fix is to dispatch those callbacks (future listeners) to a different
thread pool, not sure which one though.
If I would do a .NET-only fix, I would use the default thread pool (non
Ignite-specific), for Java-side there is no such thing as I understand.

Yes, let's have a ticket to track the issue.

On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vo...@gmail.com> wrote:

> Ilya, Pavel,
>
> Do we a have a proposal how to fix the root cause of the problem?
> Should we a have a ticket for it?
>
> ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <il...@gmail.com>:
> >
> > Hello!
> >
> > I think we should definitely stop running futures out of striped pool,
> > while holding any cache logs (stripe thread counts as one).
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <pt...@apache.org>:
> >
> > > Yes, this can be done purely on .NET side, which is an option that I
> > > consider.
> > > However, the root problem is on Java side, and I believe that we
> should fix
> > > the root problem.
> > >
> > > > violate some of Ignite assumptions: that we never run user code from
> > > certain thread pools
> > > We actually do run user code from Ignite thread pools:
> > >
> > > cache.getAsync(1).listen(fut ->
> > >     System.out.println("Get operation completed [value=" + fut.get() +
> > > ']'));
> > >
> > > `println` here is executed on the striped pool. This is stated in the
> > > docs that I linked above.
> > >
> > > Users have to be aware of this and they have to be very careful with
> > > every future listener. IMO, this is a tricky gotcha and a bad
> > > usability.
> > >
> > >
> > > Thoughts?
> > >
> > >
> > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > + dev@
> > > >
> > > > I think the current behavior, where .Net callbacks may be run from
> > > striped
> > > > pool, violate some of Ignite assumptions: that we never run user code
> > > from
> > > > certain thread pools (like sys-stripe) and that we try to limit
> options
> > > of
> > > > running user-supplied code from our internals.
> > > >
> > > > I think that future versions of .Net integration should remove the
> > > ability
> > > > of async callbacks to be called from non-user threads, even if it can
> > > lead
> > > > to performance degradation in some cases. I suggest removing this
> mode,
> > > if
> > > > possible, while keeping only the safe one, where internal threads
> are not
> > > > waiting upon completion of user code.
> > > >
> > > > In this case my issue IGNITE-12033 could be used to track this work.
> > > >
> > > > WDYT?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <pt...@apache.org>:
> > > >
> > > >> Sorry guys, I've completely missed this thread, and the topic is
> very
> > > >> important.
> > > >>
> > > >> First, a simple fix for the given example. Add the following on the
> > > first
> > > >> line of Main:
> > > >> SynchronizationContext.SetSynchronizationContext(new
> > > >> ThreadPoolSynchronizationContext());
> > > >>
> > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > >> {
> > > >>     // No-op.
> > > >> }
> > > >>
> > > >>
> > > >> Now, detailed explanation. The problem exists forever in Ignite and
> is
> > > >> mentioned in the docs briefly [1].
> > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > >>
> > > >> Breakdown:
> > > >> * Ignite (Java side) runs async callbacks (continuations) on system
> > > >> threads, and those threads have limitations (you should not call
> Ignite
> > > >> APIs from them in general)
> > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > >> * Usually `await ...` call in .NET will continue execution on the
> > > >> original Thread (simply put, actually it is more complex), so Ignite
> > > system
> > > >> thread issue is avoided
> > > >> * However, Console applications have no `SynchronizationContext`,
> so the
> > > >> continuation can't be dispatched to original thread, and is
> executed on
> > > >> current (Ignite) thread
> > > >> * Setting custom SynchronizationContext fixes the issue: all async
> > > >> continuations will be dispatched to .NET thread pool and never
> executed
> > > on
> > > >> Ignite threads
> > > >>
> > > >> However, dispatching callbacks to a different thread causes
> performance
> > > >> hit, and Ignite favors performance over usability right now.
> > > >> So it is up to the user to configure desired behavior.
> > > >>
> > > >> Let me know if you need more details.
> > > >>
> > > >> Thanks
> > > >>
> > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > > >>
> > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > ilya.kasnacheev@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hello!
> > > >>>
> > > >>> I have filed a ticket about this issue so it won't get lost.
> > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > >>>
> > > >>> Regards,
> > > >>> --
> > > >>> Ilya Kasnacheev
> > > >>>
> > > >>>
> > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > > james.prince@uk.bnpparibas.com
> > > >>> >:
> > > >>>
> > > >>>> Thanks for the response Ilya. Did you get a chance to look at this
> > > >>>> Pavel?
> > > >>>> Thanks.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > > >>>>
> > > >>>
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Павлухин Иван <vo...@gmail.com>.

Ilya, Pavel,

Do we a have a proposal how to fix the root cause of the problem?
Should we a have a ticket for it?

ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <il...@gmail.com>:
>
> Hello!
>
> I think we should definitely stop running futures out of striped pool,
> while holding any cache logs (stripe thread counts as one).
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <pt...@apache.org>:
>
> > Yes, this can be done purely on .NET side, which is an option that I
> > consider.
> > However, the root problem is on Java side, and I believe that we should fix
> > the root problem.
> >
> > > violate some of Ignite assumptions: that we never run user code from
> > certain thread pools
> > We actually do run user code from Ignite thread pools:
> >
> > cache.getAsync(1).listen(fut ->
> >     System.out.println("Get operation completed [value=" + fut.get() +
> > ']'));
> >
> > `println` here is executed on the striped pool. This is stated in the
> > docs that I linked above.
> >
> > Users have to be aware of this and they have to be very careful with
> > every future listener. IMO, this is a tricky gotcha and a bad
> > usability.
> >
> >
> > Thoughts?
> >
> >
> > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <ilya.kasnacheev@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > + dev@
> > >
> > > I think the current behavior, where .Net callbacks may be run from
> > striped
> > > pool, violate some of Ignite assumptions: that we never run user code
> > from
> > > certain thread pools (like sys-stripe) and that we try to limit options
> > of
> > > running user-supplied code from our internals.
> > >
> > > I think that future versions of .Net integration should remove the
> > ability
> > > of async callbacks to be called from non-user threads, even if it can
> > lead
> > > to performance degradation in some cases. I suggest removing this mode,
> > if
> > > possible, while keeping only the safe one, where internal threads are not
> > > waiting upon completion of user code.
> > >
> > > In this case my issue IGNITE-12033 could be used to track this work.
> > >
> > > WDYT?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <pt...@apache.org>:
> > >
> > >> Sorry guys, I've completely missed this thread, and the topic is very
> > >> important.
> > >>
> > >> First, a simple fix for the given example. Add the following on the
> > first
> > >> line of Main:
> > >> SynchronizationContext.SetSynchronizationContext(new
> > >> ThreadPoolSynchronizationContext());
> > >>
> > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > >> {
> > >>     // No-op.
> > >> }
> > >>
> > >>
> > >> Now, detailed explanation. The problem exists forever in Ignite and is
> > >> mentioned in the docs briefly [1].
> > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > >>
> > >> Breakdown:
> > >> * Ignite (Java side) runs async callbacks (continuations) on system
> > >> threads, and those threads have limitations (you should not call Ignite
> > >> APIs from them in general)
> > >> * Ignite.NET wraps async operations into native .NET Tasks
> > >> * Usually `await ...` call in .NET will continue execution on the
> > >> original Thread (simply put, actually it is more complex), so Ignite
> > system
> > >> thread issue is avoided
> > >> * However, Console applications have no `SynchronizationContext`, so the
> > >> continuation can't be dispatched to original thread, and is executed on
> > >> current (Ignite) thread
> > >> * Setting custom SynchronizationContext fixes the issue: all async
> > >> continuations will be dispatched to .NET thread pool and never executed
> > on
> > >> Ignite threads
> > >>
> > >> However, dispatching callbacks to a different thread causes performance
> > >> hit, and Ignite favors performance over usability right now.
> > >> So it is up to the user to configure desired behavior.
> > >>
> > >> Let me know if you need more details.
> > >>
> > >> Thanks
> > >>
> > >> [1] https://apacheignite.readme.io/docs/async-support
> > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > >>
> > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > ilya.kasnacheev@gmail.com>
> > >> wrote:
> > >>
> > >>> Hello!
> > >>>
> > >>> I have filed a ticket about this issue so it won't get lost.
> > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > >>>
> > >>> Regards,
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>>
> > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > james.prince@uk.bnpparibas.com
> > >>> >:
> > >>>
> > >>>> Thanks for the response Ilya. Did you get a chance to look at this
> > >>>> Pavel?
> > >>>> Thanks.
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > >>>>
> > >>>
> >



-- 
Best regards,
Ivan Pavlukhin

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Ilya Kasnacheev <il...@gmail.com>.

Hello!

I think we should definitely stop running futures out of striped pool,
while holding any cache logs (stripe thread counts as one).

Regards,
-- 
Ilya Kasnacheev


ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <pt...@apache.org>:

> Yes, this can be done purely on .NET side, which is an option that I
> consider.
> However, the root problem is on Java side, and I believe that we should fix
> the root problem.
>
> > violate some of Ignite assumptions: that we never run user code from
> certain thread pools
> We actually do run user code from Ignite thread pools:
>
> cache.getAsync(1).listen(fut ->
>     System.out.println("Get operation completed [value=" + fut.get() +
> ']'));
>
> `println` here is executed on the striped pool. This is stated in the
> docs that I linked above.
>
> Users have to be aware of this and they have to be very careful with
> every future listener. IMO, this is a tricky gotcha and a bad
> usability.
>
>
> Thoughts?
>
>
> On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <ilya.kasnacheev@gmail.com
> >
> wrote:
>
> > Hello!
> >
> > + dev@
> >
> > I think the current behavior, where .Net callbacks may be run from
> striped
> > pool, violate some of Ignite assumptions: that we never run user code
> from
> > certain thread pools (like sys-stripe) and that we try to limit options
> of
> > running user-supplied code from our internals.
> >
> > I think that future versions of .Net integration should remove the
> ability
> > of async callbacks to be called from non-user threads, even if it can
> lead
> > to performance degradation in some cases. I suggest removing this mode,
> if
> > possible, while keeping only the safe one, where internal threads are not
> > waiting upon completion of user code.
> >
> > In this case my issue IGNITE-12033 could be used to track this work.
> >
> > WDYT?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <pt...@apache.org>:
> >
> >> Sorry guys, I've completely missed this thread, and the topic is very
> >> important.
> >>
> >> First, a simple fix for the given example. Add the following on the
> first
> >> line of Main:
> >> SynchronizationContext.SetSynchronizationContext(new
> >> ThreadPoolSynchronizationContext());
> >>
> >> And put the ThreadPoolSynchronizationContext class somewhere:
> >> class ThreadPoolSynchronizationContext : SynchronizationContext
> >> {
> >>     // No-op.
> >> }
> >>
> >>
> >> Now, detailed explanation. The problem exists forever in Ignite and is
> >> mentioned in the docs briefly [1].
> >> Also mentioned in .NET docs (I've updated them a bit) [2].
> >>
> >> Breakdown:
> >> * Ignite (Java side) runs async callbacks (continuations) on system
> >> threads, and those threads have limitations (you should not call Ignite
> >> APIs from them in general)
> >> * Ignite.NET wraps async operations into native .NET Tasks
> >> * Usually `await ...` call in .NET will continue execution on the
> >> original Thread (simply put, actually it is more complex), so Ignite
> system
> >> thread issue is avoided
> >> * However, Console applications have no `SynchronizationContext`, so the
> >> continuation can't be dispatched to original thread, and is executed on
> >> current (Ignite) thread
> >> * Setting custom SynchronizationContext fixes the issue: all async
> >> continuations will be dispatched to .NET thread pool and never executed
> on
> >> Ignite threads
> >>
> >> However, dispatching callbacks to a different thread causes performance
> >> hit, and Ignite favors performance over usability right now.
> >> So it is up to the user to configure desired behavior.
> >>
> >> Let me know if you need more details.
> >>
> >> Thanks
> >>
> >> [1] https://apacheignite.readme.io/docs/async-support
> >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> >>
> >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>
> >> wrote:
> >>
> >>> Hello!
> >>>
> >>> I have filed a ticket about this issue so it won't get lost.
> >>> https://issues.apache.org/jira/browse/IGNITE-12033
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> james.prince@uk.bnpparibas.com
> >>> >:
> >>>
> >>>> Thanks for the response Ilya. Did you get a chance to look at this
> >>>> Pavel?
> >>>> Thanks.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >>>>
> >>>
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Pavel Tupitsyn <pt...@apache.org>.

Yes, this can be done purely on .NET side, which is an option that I
consider.
However, the root problem is on Java side, and I believe that we should fix
the root problem.

> violate some of Ignite assumptions: that we never run user code from
certain thread pools
We actually do run user code from Ignite thread pools:

cache.getAsync(1).listen(fut ->
    System.out.println("Get operation completed [value=" + fut.get() + ']'));

`println` here is executed on the striped pool. This is stated in the
docs that I linked above.

Users have to be aware of this and they have to be very careful with
every future listener. IMO, this is a tricky gotcha and a bad
usability.


Thoughts?


On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <il...@gmail.com>
wrote:

> Hello!
>
> + dev@
>
> I think the current behavior, where .Net callbacks may be run from striped
> pool, violate some of Ignite assumptions: that we never run user code from
> certain thread pools (like sys-stripe) and that we try to limit options of
> running user-supplied code from our internals.
>
> I think that future versions of .Net integration should remove the ability
> of async callbacks to be called from non-user threads, even if it can lead
> to performance degradation in some cases. I suggest removing this mode, if
> possible, while keeping only the safe one, where internal threads are not
> waiting upon completion of user code.
>
> In this case my issue IGNITE-12033 could be used to track this work.
>
> WDYT?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <pt...@apache.org>:
>
>> Sorry guys, I've completely missed this thread, and the topic is very
>> important.
>>
>> First, a simple fix for the given example. Add the following on the first
>> line of Main:
>> SynchronizationContext.SetSynchronizationContext(new
>> ThreadPoolSynchronizationContext());
>>
>> And put the ThreadPoolSynchronizationContext class somewhere:
>> class ThreadPoolSynchronizationContext : SynchronizationContext
>> {
>>     // No-op.
>> }
>>
>>
>> Now, detailed explanation. The problem exists forever in Ignite and is
>> mentioned in the docs briefly [1].
>> Also mentioned in .NET docs (I've updated them a bit) [2].
>>
>> Breakdown:
>> * Ignite (Java side) runs async callbacks (continuations) on system
>> threads, and those threads have limitations (you should not call Ignite
>> APIs from them in general)
>> * Ignite.NET wraps async operations into native .NET Tasks
>> * Usually `await ...` call in .NET will continue execution on the
>> original Thread (simply put, actually it is more complex), so Ignite system
>> thread issue is avoided
>> * However, Console applications have no `SynchronizationContext`, so the
>> continuation can't be dispatched to original thread, and is executed on
>> current (Ignite) thread
>> * Setting custom SynchronizationContext fixes the issue: all async
>> continuations will be dispatched to .NET thread pool and never executed on
>> Ignite threads
>>
>> However, dispatching callbacks to a different thread causes performance
>> hit, and Ignite favors performance over usability right now.
>> So it is up to the user to configure desired behavior.
>>
>> Let me know if you need more details.
>>
>> Thanks
>>
>> [1] https://apacheignite.readme.io/docs/async-support
>> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
>>
>> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <il...@gmail.com>
>> wrote:
>>
>>> Hello!
>>>
>>> I have filed a ticket about this issue so it won't get lost.
>>> https://issues.apache.org/jira/browse/IGNITE-12033
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <james.prince@uk.bnpparibas.com
>>> >:
>>>
>>>> Thanks for the response Ilya. Did you get a chance to look at this
>>>> Pavel?
>>>> Thanks.
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Ilya Kasnacheev <il...@gmail.com>.

Hello!

+ dev@

I think the current behavior, where .Net callbacks may be run from striped
pool, violate some of Ignite assumptions: that we never run user code from
certain thread pools (like sys-stripe) and that we try to limit options of
running user-supplied code from our internals.

I think that future versions of .Net integration should remove the ability
of async callbacks to be called from non-user threads, even if it can lead
to performance degradation in some cases. I suggest removing this mode, if
possible, while keeping only the safe one, where internal threads are not
waiting upon completion of user code.

In this case my issue IGNITE-12033 could be used to track this work.

WDYT?

Regards,
-- 
Ilya Kasnacheev


ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <pt...@apache.org>:

> Sorry guys, I've completely missed this thread, and the topic is very
> important.
>
> First, a simple fix for the given example. Add the following on the first
> line of Main:
> SynchronizationContext.SetSynchronizationContext(new
> ThreadPoolSynchronizationContext());
>
> And put the ThreadPoolSynchronizationContext class somewhere:
> class ThreadPoolSynchronizationContext : SynchronizationContext
> {
>     // No-op.
> }
>
>
> Now, detailed explanation. The problem exists forever in Ignite and is
> mentioned in the docs briefly [1].
> Also mentioned in .NET docs (I've updated them a bit) [2].
>
> Breakdown:
> * Ignite (Java side) runs async callbacks (continuations) on system
> threads, and those threads have limitations (you should not call Ignite
> APIs from them in general)
> * Ignite.NET wraps async operations into native .NET Tasks
> * Usually `await ...` call in .NET will continue execution on the original
> Thread (simply put, actually it is more complex), so Ignite system thread
> issue is avoided
> * However, Console applications have no `SynchronizationContext`, so the
> continuation can't be dispatched to original thread, and is executed on
> current (Ignite) thread
> * Setting custom SynchronizationContext fixes the issue: all async
> continuations will be dispatched to .NET thread pool and never executed on
> Ignite threads
>
> However, dispatching callbacks to a different thread causes performance
> hit, and Ignite favors performance over usability right now.
> So it is up to the user to configure desired behavior.
>
> Let me know if you need more details.
>
> Thanks
>
> [1] https://apacheignite.readme.io/docs/async-support
> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
>
> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> I have filed a ticket about this issue so it won't get lost.
>> https://issues.apache.org/jira/browse/IGNITE-12033
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <james.prince@uk.bnpparibas.com
>> >:
>>
>>> Thanks for the response Ilya. Did you get a chance to look at this Pavel?
>>> Thanks.
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Maxim Muzafarov <mm...@apache.org>.

Igniters,

Bump this thread up. It seems the issue [1] is important but still
unassigned.


[1] https://issues.apache.org/jira/browse/IGNITE-12033

On Thu, 15 Aug 2019 at 15:01, Pavel Tupitsyn <pt...@apache.org> wrote:

> Hi Eduard,
>
> Yes, that is the same issue.
> The workaround above is not enough, because SetSynchronizationContext
> affects only the current thread.
> After the first async operation completes, the continuation is dispatched
> to some ThreadPool thread,
> which may not have a SynchronizationContext set.
>
> Updated workaround:
>
> class ThreadPoolSynchronizationContext : SynchronizationContext
> {
>     public override void Post(SendOrPostCallback d, object state)
>     {
>         ThreadPool.QueueUserWorkItem(s =>
>         {
>             SetSynchronizationContext(this);
>             d(s);
>         }, state);
>     }
>
>
> Bear in mind, though - this can affect your entire application performance.
>
> We are discussing Ignite-wide fix for this on the dev list.
>
>
>
> On Wed, Aug 14, 2019 at 1:21 PM e.llull <ed...@llull.net> wrote:
>
>> Hi guys,
>>
>> We are also facing a similar problem, if not the same. Our main difference
>> with the initial reproducer is that we are using the Thick Client. We
>> applied the suggested fix of setting the SynchronizationContext, but we
>> also
>> perform a GetAsync after the initial PutAsync. Also, I added a loop around
>> the Replace because sometimes it takes several iterations to block.
>>
>> Here is the code:
>> using System;
>> using System.Collections.Generic;
>> using System.Threading;
>> using System.Threading.Tasks;
>> using Apache.Ignite.Core;
>> using Apache.Ignite.Core.Cache.Configuration;
>> using Apache.Ignite.Core.Transactions;
>>
>> namespace IgniteHangTest
>> {
>>     class Program : IDisposable
>>     {
>>         protected readonly IIgnite server;
>>
>>         protected readonly IIgnite client;
>>
>>         public static async Task Main(string[] args)
>>         {
>>             SynchronizationContext.SetSynchronizationContext(new
>> ThreadPoolSynchronizationContext());
>>
>>             using (var program = new Program())
>>             {
>>                 await program.Run();
>>             }
>>         }
>>
>>         public Program() {
>>             server = Ignition.Start(IgniteConfiguration("server"));
>>             server.GetOrCreateCache<int, string>(new
>> CacheConfiguration("TestCache")
>>             {
>>                 AtomicityMode = CacheAtomicityMode.Transactional,
>>
>>             });
>>
>>             var clientConfiguration = IgniteConfiguration("client");
>>             clientConfiguration.ClientMode = true;
>>             client = Ignition.Start(clientConfiguration);
>>         }
>>
>>         private async Task Run() {
>>             var cache = client.GetCache<int, string>("TestCache");
>>
>>             Console.WriteLine("Put initial value");
>>             await cache.PutAsync(0, "Test");
>>
>>             Console.WriteLine("Get initial value");
>>             string initialValue = await cache.GetAsync(0);  // if removed,
>> it works
>>
>>             Console.WriteLine("Entering Replace loop");
>>             for(int i = 0; i < 100; i++)
>>             {
>>                 cache.Replace(0, "Replace " + i);  // It blocks here
>>                 Console.WriteLine("Loop: i = {0}", i);
>>             }
>>
>>             Console.WriteLine("End");
>>         }
>>
>>         public void Dispose() {
>>             Ignition.Stop("client", true);
>>             Ignition.Stop("server", true);
>>         }
>>
>>         private IgniteConfiguration IgniteConfiguration(string
>> instanceName)
>>         {
>>             return new IgniteConfiguration
>>             {
>>                 IgniteInstanceName = instanceName,
>>                 JvmOptions = new List<string> { "-DIGNITE_QUIET=false",
>> },
>>                 TransactionConfiguration = new TransactionConfiguration
>>                 {
>>                     DefaultTimeout = TimeSpan.FromSeconds(5),
>>                     DefaultTransactionConcurrency =
>> TransactionConcurrency.Optimistic,
>>                     DefaultTransactionIsolation =
>> TransactionIsolation.Serializable
>>                 },
>>             };
>>         }
>>     }
>>
>>     class ThreadPoolSynchronizationContext : SynchronizationContext { }
>> }
>>
>> In the reproducer we are starting two Ignite nodes, one as the server and
>> one with ClientMode = true. This is only in the reproducer, in the real
>> use
>> case the server Ignite node is started in a different machine but the
>> problem also arises with the "external" server Ignite node.
>>
>> If the line `string initialValue = await cache.GetAsync(0);` is removed
>> the
>> programs finishes successfully.
>>
>> In the console, the relevant logs are:
>>
>> Critical system error detected. Will be handled accordingly to configured
>> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
>> super=AbstractFailureHandler
>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
>> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
>> o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
>> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
>> class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
>> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
>>         at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
>>
>>         at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
>>
>>         at
>> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
>>
>>         at
>> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
>>
>>         at
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
>>
>>         at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>>
>>         at java.lang.Thread.run(Thread.java:748)
>>
>> And the stack trace of the sys-stripe-6 thread is:
>> Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
>> waitCnt=11]
>>         at sun.misc.Unsafe.park(Native Method)
>>         at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
>>         at
>> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
>>         at
>> o.a.i.i.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896)
>>
>>         at
>> o.a.i.i.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294)
>>
>>         at
>> o.a.i.i.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012)
>>
>>         at
>> o.a.i.i.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483)
>>
>>         at
>> o.a.i.i.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
>>
>>         at
>>
>> o.a.i.i.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
>> Method)
>>         at
>> o.a.i.i.processors.platform.callback.PlatformCallbackGateway.futureObjectResult(PlatformCallbackGateway.java:626)
>>
>>         at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:219)
>>
>>         at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189)
>>
>>         at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382)
>>
>>         at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377)
>>
>>         at
>> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215)
>>
>>         at
>> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179)
>>
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
>>
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
>>
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
>>         at
>> o.a.i.i.processors.cache.GridCacheFutureAdapter.onDone(GridCacheFutureAdapter.java:55)
>>
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
>>         at
>> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onDone(GridPartitionedSingleGetFuture.java:774)
>>
>>         at
>> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
>>         at
>> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:696)
>>
>>         at
>> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:551)
>>
>>         at
>> o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:368)
>>
>>         at
>> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:87)
>>
>>         at
>> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:132)
>>
>>         at
>> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:130)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
>>
>>         at
>> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
>>
>>         at
>> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>
>>         at
>> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>
>>         at
>> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>
>>         at
>> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>
>>         at
>> o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
>>         at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Pavel Tupitsyn <pt...@apache.org>.

Hi Eduard,

Yes, that is the same issue.
The workaround above is not enough, because SetSynchronizationContext
affects only the current thread.
After the first async operation completes, the continuation is dispatched
to some ThreadPool thread,
which may not have a SynchronizationContext set.

Updated workaround:

class ThreadPoolSynchronizationContext : SynchronizationContext
{
    public override void Post(SendOrPostCallback d, object state)
    {
        ThreadPool.QueueUserWorkItem(s =>
        {
            SetSynchronizationContext(this);
            d(s);
        }, state);
    }


Bear in mind, though - this can affect your entire application performance.

We are discussing Ignite-wide fix for this on the dev list.



On Wed, Aug 14, 2019 at 1:21 PM e.llull <ed...@llull.net> wrote:

> Hi guys,
>
> We are also facing a similar problem, if not the same. Our main difference
> with the initial reproducer is that we are using the Thick Client. We
> applied the suggested fix of setting the SynchronizationContext, but we
> also
> perform a GetAsync after the initial PutAsync. Also, I added a loop around
> the Replace because sometimes it takes several iterations to block.
>
> Here is the code:
> using System;
> using System.Collections.Generic;
> using System.Threading;
> using System.Threading.Tasks;
> using Apache.Ignite.Core;
> using Apache.Ignite.Core.Cache.Configuration;
> using Apache.Ignite.Core.Transactions;
>
> namespace IgniteHangTest
> {
>     class Program : IDisposable
>     {
>         protected readonly IIgnite server;
>
>         protected readonly IIgnite client;
>
>         public static async Task Main(string[] args)
>         {
>             SynchronizationContext.SetSynchronizationContext(new
> ThreadPoolSynchronizationContext());
>
>             using (var program = new Program())
>             {
>                 await program.Run();
>             }
>         }
>
>         public Program() {
>             server = Ignition.Start(IgniteConfiguration("server"));
>             server.GetOrCreateCache<int, string>(new
> CacheConfiguration("TestCache")
>             {
>                 AtomicityMode = CacheAtomicityMode.Transactional,
>
>             });
>
>             var clientConfiguration = IgniteConfiguration("client");
>             clientConfiguration.ClientMode = true;
>             client = Ignition.Start(clientConfiguration);
>         }
>
>         private async Task Run() {
>             var cache = client.GetCache<int, string>("TestCache");
>
>             Console.WriteLine("Put initial value");
>             await cache.PutAsync(0, "Test");
>
>             Console.WriteLine("Get initial value");
>             string initialValue = await cache.GetAsync(0);  // if removed,
> it works
>
>             Console.WriteLine("Entering Replace loop");
>             for(int i = 0; i < 100; i++)
>             {
>                 cache.Replace(0, "Replace " + i);  // It blocks here
>                 Console.WriteLine("Loop: i = {0}", i);
>             }
>
>             Console.WriteLine("End");
>         }
>
>         public void Dispose() {
>             Ignition.Stop("client", true);
>             Ignition.Stop("server", true);
>         }
>
>         private IgniteConfiguration IgniteConfiguration(string
> instanceName)
>         {
>             return new IgniteConfiguration
>             {
>                 IgniteInstanceName = instanceName,
>                 JvmOptions = new List<string> { "-DIGNITE_QUIET=false", },
>                 TransactionConfiguration = new TransactionConfiguration
>                 {
>                     DefaultTimeout = TimeSpan.FromSeconds(5),
>                     DefaultTransactionConcurrency =
> TransactionConcurrency.Optimistic,
>                     DefaultTransactionIsolation =
> TransactionIsolation.Serializable
>                 },
>             };
>         }
>     }
>
>     class ThreadPoolSynchronizationContext : SynchronizationContext { }
> }
>
> In the reproducer we are starting two Ignite nodes, one as the server and
> one with ClientMode = true. This is only in the reproducer, in the real use
> case the server Ignite node is started in a different machine but the
> problem also arises with the "external" server Ignite node.
>
> If the line `string initialValue = await cache.GetAsync(0);` is removed the
> programs finishes successfully.
>
> In the console, the relevant logs are:
>
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
> class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
>         at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
>
>         at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
>
>         at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
>
>         at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
>
>         at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
>
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>         at java.lang.Thread.run(Thread.java:748)
>
> And the stack trace of the sys-stripe-6 thread is:
> Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
> waitCnt=11]
>         at sun.misc.Unsafe.park(Native Method)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>         at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
>         at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
>         at
> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470)
>
>         at
> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468)
>
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233)
>
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468)
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
>         at
> o.a.i.i.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896)
>
>         at
> o.a.i.i.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294)
>
>         at
> o.a.i.i.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012)
>
>         at
> o.a.i.i.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483)
>
>         at
> o.a.i.i.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
>
>         at
>
> o.a.i.i.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
> Method)
>         at
> o.a.i.i.processors.platform.callback.PlatformCallbackGateway.futureObjectResult(PlatformCallbackGateway.java:626)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:219)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382)
>
>         at
> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377)
>
>         at
> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215)
>
>         at
> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
>         at
> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
>         at
> o.a.i.i.processors.cache.GridCacheFutureAdapter.onDone(GridCacheFutureAdapter.java:55)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
>         at
> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onDone(GridPartitionedSingleGetFuture.java:774)
>
>         at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
>         at
> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:696)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:551)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:368)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:87)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:132)
>
>         at
> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:130)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
>
>         at
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
>
>         at
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>
>         at
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>
>         at
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>
>         at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>
>         at
> o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
>         at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by "e.llull" <ed...@llull.net>.

Hi guys, 

We are also facing a similar problem, if not the same. Our main difference
with the initial reproducer is that we are using the Thick Client. We
applied the suggested fix of setting the SynchronizationContext, but we also
perform a GetAsync after the initial PutAsync. Also, I added a loop around
the Replace because sometimes it takes several iterations to block. 

Here is the code: 
using System; 
using System.Collections.Generic; 
using System.Threading; 
using System.Threading.Tasks; 
using Apache.Ignite.Core; 
using Apache.Ignite.Core.Cache.Configuration; 
using Apache.Ignite.Core.Transactions; 

namespace IgniteHangTest 
{ 
    class Program : IDisposable 
    { 
        protected readonly IIgnite server; 

        protected readonly IIgnite client; 

        public static async Task Main(string[] args) 
        { 
            SynchronizationContext.SetSynchronizationContext(new
ThreadPoolSynchronizationContext()); 

            using (var program = new Program()) 
            { 
                await program.Run(); 
            } 
        } 

        public Program() { 
            server = Ignition.Start(IgniteConfiguration("server")); 
            server.GetOrCreateCache<int, string>(new
CacheConfiguration("TestCache") 
            { 
                AtomicityMode = CacheAtomicityMode.Transactional,                 
            }); 

            var clientConfiguration = IgniteConfiguration("client"); 
            clientConfiguration.ClientMode = true; 
            client = Ignition.Start(clientConfiguration); 
        } 

        private async Task Run() { 
            var cache = client.GetCache<int, string>("TestCache"); 

            Console.WriteLine("Put initial value"); 
            await cache.PutAsync(0, "Test"); 

            Console.WriteLine("Get initial value"); 
            string initialValue = await cache.GetAsync(0);  // if removed,
it works 

            Console.WriteLine("Entering Replace loop"); 
            for(int i = 0; i < 100; i++) 
            { 
                cache.Replace(0, "Replace " + i);  // It blocks here 
                Console.WriteLine("Loop: i = {0}", i); 
            } 

            Console.WriteLine("End"); 
        } 

        public void Dispose() { 
            Ignition.Stop("client", true); 
            Ignition.Stop("server", true); 
        } 

        private IgniteConfiguration IgniteConfiguration(string instanceName) 
        { 
            return new IgniteConfiguration 
            { 
                IgniteInstanceName = instanceName, 
                JvmOptions = new List<string> { "-DIGNITE_QUIET=false", }, 
                TransactionConfiguration = new TransactionConfiguration 
                { 
                    DefaultTimeout = TimeSpan.FromSeconds(5), 
                    DefaultTransactionConcurrency =
TransactionConcurrency.Optimistic, 
                    DefaultTransactionIsolation =
TransactionIsolation.Serializable 
                }, 
            }; 
        } 
    } 

    class ThreadPoolSynchronizationContext : SynchronizationContext { } 
} 

In the reproducer we are starting two Ignite nodes, one as the server and
one with ClientMode = true. This is only in the reproducer, in the real use
case the server Ignite node is started in a different machine but the
problem also arises with the "external" server Ignite node. 

If the line `string initialValue = await cache.GetAsync(0);` is removed the
programs finishes successfully. 

In the console, the relevant logs are: 

Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]] 
class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
igniteInstanceName=client, finished=false, heartbeatTs=1565776844696] 
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) 
        at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) 
        at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) 
        at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) 
        at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) 
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) 
        at java.lang.Thread.run(Thread.java:748) 

And the stack trace of the sys-stripe-6 thread is: 
Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
waitCnt=11] 
        at sun.misc.Unsafe.park(Native Method) 
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) 
        at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) 
        at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) 
        at
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470) 
        at
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468) 
        at
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233) 
        at
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468) 
        at
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449) 
        at
o.a.i.i.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896) 
        at
o.a.i.i.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294) 
        at
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012) 
        at
o.a.i.i.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483) 
        at
o.a.i.i.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67) 
        at
o.a.i.i.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
Method) 
        at
o.a.i.i.processors.platform.callback.PlatformCallbackGateway.futureObjectResult(PlatformCallbackGateway.java:626) 
        at
o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:219) 
        at
o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189) 
        at
o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382) 
        at
o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377) 
        at
o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215) 
        at
o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179) 
        at
o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) 
        at
o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349) 
        at
o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337) 
        at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497) 
        at
o.a.i.i.processors.cache.GridCacheFutureAdapter.onDone(GridCacheFutureAdapter.java:55) 
        at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476) 
        at
o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onDone(GridPartitionedSingleGetFuture.java:774) 
        at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453) 
        at
o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:696) 
        at
o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:551) 
        at
o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:368) 
        at
o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:87) 
        at
o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:132) 
        at
o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:130) 
        at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) 
        at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) 
        at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) 
        at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) 
        at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) 
        at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) 
        at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) 
        at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) 
        at
o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) 
        at
o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) 
        at
o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) 
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) 
        at java.lang.Thread.run(Thread.java:748) 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Pavel Tupitsyn <pt...@apache.org>.

Sorry guys, I've completely missed this thread, and the topic is very
important.

First, a simple fix for the given example. Add the following on the first
line of Main:
SynchronizationContext.SetSynchronizationContext(new
ThreadPoolSynchronizationContext());

And put the ThreadPoolSynchronizationContext class somewhere:
class ThreadPoolSynchronizationContext : SynchronizationContext
{
    // No-op.
}

Now, detailed explanation. The problem exists forever in Ignite and is
mentioned in the docs briefly [1].
Also mentioned in .NET docs (I've updated them a bit) [2].

Breakdown:
* Ignite (Java side) runs async callbacks (continuations) on system
threads, and those threads have limitations (you should not call Ignite
APIs from them in general)
* Ignite.NET wraps async operations into native .NET Tasks
* Usually `await ...` call in .NET will continue execution on the original
Thread (simply put, actually it is more complex), so Ignite system thread
issue is avoided
* However, Console applications have no `SynchronizationContext`, so the
continuation can't be dispatched to original thread, and is executed on
current (Ignite) thread
* Setting custom SynchronizationContext fixes the issue: all async
continuations will be dispatched to .NET thread pool and never executed on
Ignite threads

However, dispatching callbacks to a different thread causes performance
hit, and Ignite favors performance over usability right now.
So it is up to the user to configure desired behavior.

Let me know if you need more details.

Thanks

[1] https://apacheignite.readme.io/docs/async-support
[2] https://apacheignite-net.readme.io/docs/asynchronous-support

On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <il...@gmail.com>
wrote:

> Hello!
>
> I have filed a ticket about this issue so it won't get lost.
> https://issues.apache.org/jira/browse/IGNITE-12033
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 2 мая 2019 г. в 10:53, Barney Pippin <ja...@uk.bnpparibas.com>:
>
>> Thanks for the response Ilya. Did you get a chance to look at this Pavel?
>> Thanks.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

Posted by Ilya Kasnacheev <il...@gmail.com>.

Hello!

I have filed a ticket about this issue so it won't get lost.
https://issues.apache.org/jira/browse/IGNITE-12033

Regards,
-- 
Ilya Kasnacheev


чт, 2 мая 2019 г. в 10:53, Barney Pippin <ja...@uk.bnpparibas.com>:

> Thanks for the response Ilya. Did you get a chance to look at this Pavel?
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>