You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Maxim Muzafarov <mm...@apache.org> on 2020/01/13 10:11:05 UTC
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Igniters,
Bump this thread up. It seems the issue [1] is important but still
unassigned.
[1] https://issues.apache.org/jira/browse/IGNITE-12033
On Thu, 15 Aug 2019 at 15:01, Pavel Tupitsyn <pt...@apache.org> wrote:
> Hi Eduard,
>
> Yes, that is the same issue.
> The workaround above is not enough, because SetSynchronizationContext
> affects only the current thread.
> After the first async operation completes, the continuation is dispatched
> to some ThreadPool thread,
> which may not have a SynchronizationContext set.
>
> Updated workaround:
>
> class ThreadPoolSynchronizationContext : SynchronizationContext
> {
> public override void Post(SendOrPostCallback d, object state)
> {
> ThreadPool.QueueUserWorkItem(s =>
> {
> SetSynchronizationContext(this);
> d(s);
> }, state);
> }
>
>
> Bear in mind, though - this can affect your entire application performance.
>
> We are discussing Ignite-wide fix for this on the dev list.
>
>
>
> On Wed, Aug 14, 2019 at 1:21 PM e.llull <ed...@llull.net> wrote:
>
>> Hi guys,
>>
>> We are also facing a similar problem, if not the same. Our main difference
>> with the initial reproducer is that we are using the Thick Client. We
>> applied the suggested fix of setting the SynchronizationContext, but we
>> also
>> perform a GetAsync after the initial PutAsync. Also, I added a loop around
>> the Replace because sometimes it takes several iterations to block.
>>
>> Here is the code:
>> using System;
>> using System.Collections.Generic;
>> using System.Threading;
>> using System.Threading.Tasks;
>> using Apache.Ignite.Core;
>> using Apache.Ignite.Core.Cache.Configuration;
>> using Apache.Ignite.Core.Transactions;
>>
>> namespace IgniteHangTest
>> {
>> class Program : IDisposable
>> {
>> protected readonly IIgnite server;
>>
>> protected readonly IIgnite client;
>>
>> public static async Task Main(string[] args)
>> {
>> SynchronizationContext.SetSynchronizationContext(new
>> ThreadPoolSynchronizationContext());
>>
>> using (var program = new Program())
>> {
>> await program.Run();
>> }
>> }
>>
>> public Program() {
>> server = Ignition.Start(IgniteConfiguration("server"));
>> server.GetOrCreateCache<int, string>(new
>> CacheConfiguration("TestCache")
>> {
>> AtomicityMode = CacheAtomicityMode.Transactional,
>>
>> });
>>
>> var clientConfiguration = IgniteConfiguration("client");
>> clientConfiguration.ClientMode = true;
>> client = Ignition.Start(clientConfiguration);
>> }
>>
>> private async Task Run() {
>> var cache = client.GetCache<int, string>("TestCache");
>>
>> Console.WriteLine("Put initial value");
>> await cache.PutAsync(0, "Test");
>>
>> Console.WriteLine("Get initial value");
>> string initialValue = await cache.GetAsync(0); // if removed,
>> it works
>>
>> Console.WriteLine("Entering Replace loop");
>> for(int i = 0; i < 100; i++)
>> {
>> cache.Replace(0, "Replace " + i); // It blocks here
>> Console.WriteLine("Loop: i = {0}", i);
>> }
>>
>> Console.WriteLine("End");
>> }
>>
>> public void Dispose() {
>> Ignition.Stop("client", true);
>> Ignition.Stop("server", true);
>> }
>>
>> private IgniteConfiguration IgniteConfiguration(string
>> instanceName)
>> {
>> return new IgniteConfiguration
>> {
>> IgniteInstanceName = instanceName,
>> JvmOptions = new List<string> { "-DIGNITE_QUIET=false",
>> },
>> TransactionConfiguration = new TransactionConfiguration
>> {
>> DefaultTimeout = TimeSpan.FromSeconds(5),
>> DefaultTransactionConcurrency =
>> TransactionConcurrency.Optimistic,
>> DefaultTransactionIsolation =
>> TransactionIsolation.Serializable
>> },
>> };
>> }
>> }
>>
>> class ThreadPoolSynchronizationContext : SynchronizationContext { }
>> }
>>
>> In the reproducer we are starting two Ignite nodes, one as the server and
>> one with ClientMode = true. This is only in the reproducer, in the real
>> use
>> case the server Ignite node is started in a different machine but the
>> problem also arises with the "external" server Ignite node.
>>
>> If the line `string initialValue = await cache.GetAsync(0);` is removed
>> the
>> programs finishes successfully.
>>
>> In the console, the relevant logs are:
>>
>> Critical system error detected. Will be handled accordingly to configured
>> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
>> super=AbstractFailureHandler
>> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
>> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
>> o.a.i.IgniteException: GridWorker [name=sys-stripe-6,
>> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]]]
>> class org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-6,
>> igniteInstanceName=client, finished=false, heartbeatTs=1565776844696]
>> at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
>>
>> at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
>>
>> at
>> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
>>
>> at
>> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
>>
>> at
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
>>
>> at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> And the stack trace of the sys-stripe-6 thread is:
>> Thread [name="sys-stripe-6-#52%client%", id=82, state=WAITING, blockCnt=0,
>> waitCnt=11]
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
>> at
>> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
>> at
>> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
>> at
>> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2470)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2468)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4233)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2468)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
>> at
>> o.a.i.i.processors.cache.GridCacheAdapter.replace(GridCacheAdapter.java:2896)
>>
>> at
>> o.a.i.i.processors.cache.IgniteCacheProxyImpl.replace(IgniteCacheProxyImpl.java:1294)
>>
>> at
>> o.a.i.i.processors.cache.GatewayProtectedCacheProxy.replace(GatewayProtectedCacheProxy.java:1012)
>>
>> at
>> o.a.i.i.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:483)
>>
>> at
>> o.a.i.i.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
>>
>> at
>>
>> o.a.i.i.processors.platform.callback.PlatformCallbackUtils.inLongLongLongObjectOutLong(Native
>> Method)
>> at
>> o.a.i.i.processors.platform.callback.PlatformCallbackGateway.futureObjectResult(PlatformCallbackGateway.java:626)
>>
>> at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:219)
>>
>> at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$1.apply(PlatformFutureUtils.java:189)
>>
>> at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:382)
>>
>> at
>> o.a.i.i.processors.platform.utils.PlatformFutureUtils$FutureListenable$1.apply(PlatformFutureUtils.java:377)
>>
>> at
>> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:215)
>>
>> at
>> o.a.i.i.util.future.IgniteFutureImpl$InternalFutureListener.apply(IgniteFutureImpl.java:179)
>>
>> at
>> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
>>
>> at
>> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
>> at
>> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
>>
>> at
>> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
>> at
>> o.a.i.i.processors.cache.GridCacheFutureAdapter.onDone(GridCacheFutureAdapter.java:55)
>>
>> at
>> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
>> at
>> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onDone(GridPartitionedSingleGetFuture.java:774)
>>
>> at
>> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
>> at
>> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:696)
>>
>> at
>> o.a.i.i.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:551)
>>
>> at
>> o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:368)
>>
>> at
>> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:87)
>>
>> at
>> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:132)
>>
>> at
>> o.a.i.i.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$2.apply(GridDhtColocatedCache.java:130)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
>>
>> at
>> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
>>
>> at
>> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>
>> at
>> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>
>> at
>> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>
>> at
>> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>
>> at
>> o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
>> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>