You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Alexey Scherbakov (Jira)" <ji...@apache.org> on 2021/05/11 13:38:00 UTC
[jira] [Comment Edited] (IGNITE-14583) Server node fails on remote
filter deployment after client reconnect with new node id
[ https://issues.apache.org/jira/browse/IGNITE-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342570#comment-17342570 ]
Alexey Scherbakov edited comment on IGNITE-14583 at 5/11/21, 1:37 PM:
----------------------------------------------------------------------
[~ibelyakov]
Overall the changes look good, but I've noticed the test [1] has become flaky due to triggered FH with following stack trace [2]
Can you fix it ?
[1] org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryOperationP2PTest#testTxReplicatedClient
[2]
{noformat}
2021-05-11 16:31:44,988][ERROR][sys-#283%continuous.CacheContinuousQueryOperationP2PTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Failed to send custom message: client is starting.]]
class org.apache.ignite.IgniteCheckedException: Failed to send custom message: client is starting.
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.sendCustomEvent(GridDiscoveryManager.java:2251)
at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl.startWrite(DistributedMetaStorageImpl.java:1071)
at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl.writeAsync(DistributedMetaStorageImpl.java:482)
at org.apache.ignite.internal.processors.cluster.ClusterProcessor.lambda$onReadyForWrite$3(ClusterProcessor.java:289)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7181)
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:829)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Failed to send custom message: client is starting.
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendCustomEvent(ClientImpl.java:514)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.sendCustomEvent(TcpDiscoverySpi.java:546)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.sendCustomEvent(GridDiscoveryManager.java:2243)
... 9 more{noformat}
was (Author: ascherbakov):
[~ibelyakov]
Overall the changes look good, but I've noticed the test [1] has become flaky due to triggered FH with following stack trace [2]
Can you fix it ?
[1] org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryOperationP2PTest#testTxReplicatedClient
[2]
{noformat}
[2021-05-11 16:31:44,988][ERROR][sys-#283%continuous.CacheContinuousQueryOperationP2PTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Failed to send custom message: client is starting.]][2021-05-11 16:31:44,988][ERROR][sys-#283%continuous.CacheContinuousQueryOperationP2PTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Failed to send custom message: client is starting.]]class org.apache.ignite.IgniteCheckedException: Failed to send custom message: client is starting. at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.sendCustomEvent(GridDiscoveryManager.java:2251) at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl.startWrite(DistributedMetaStorageImpl.java:1071) at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl.writeAsync(DistributedMetaStorageImpl.java:482) at org.apache.ignite.internal.processors.cluster.ClusterProcessor.lambda$onReadyForWrite$3(ClusterProcessor.java:289) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7181) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:829) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: class org.apache.ignite.IgniteException: Failed to send custom message: client is starting. at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendCustomEvent(ClientImpl.java:514) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.sendCustomEvent(TcpDiscoverySpi.java:546) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.sendCustomEvent(GridDiscoveryManager.java:2243) ... 9 more {noformat}
> Server node fails on remote filter deployment after client reconnect with new node id
> -------------------------------------------------------------------------------------
>
> Key: IGNITE-14583
> URL: https://issues.apache.org/jira/browse/IGNITE-14583
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.10
> Reporter: Igor Belyakov
> Assignee: Igor Belyakov
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Cluster contains 1 server and 1 client node. PeerClassLoading turned on.
> *1.* Client node deploys CQ with remote filter, server node doesn't have classes required for remote filter in classpath.
> *2.* Remote filter successfully deployed to server node.
> *3.* Client node disconnects and reconnects by using new node id:
> [15:44:43] Client node was reconnected after it was already considered failed by the server topology (this could happen after all servers restarted or due to a long network outage between the client and servers). All continuous queries and remote event listeners created by this client will be unsubscribed, consider listening to EVT_CLIENT_NODE_RECONNECTED event to restore them.
> *4.* On connect, client node tries to deploy remote filter to server node, but server node fails due to wrong ClassLoaderId, which is sent in GridDeploymentInfo (used ClassLoaderId for old client node):
> {code:java}
> [15:44:42] (err) Failed to notify listener: o.a.i.i.util.future.GridFutureChainListener@778d0fc6class org.apache.ignite.IgniteException: Failed to initialize a remote filter.
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.lambda$register$2bf956f5$1(CacheContinuousQueryHandler.java:353)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:77)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:69)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:29)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:398)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:510)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:489)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:477)
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.p2pUnmarshal(CacheContinuousQueryHandler.java:1311)
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.p2pUnmarshal(CacheContinuousQueryHandler.java:1283)
> at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$8.run(GridContinuousProcessor.java:693)
> at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7162)
> at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:826)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: class org.apache.ignite.internal.IgniteDeploymentCheckedException: Failed to obtain deployment for class: Client1$$Lambda$513/0x00000008003b8840
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryDeployableObject.unmarshal(CacheContinuousQueryDeployableObject.java:94)
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.p2pUnmarshal(CacheContinuousQueryHandler.java:1308)
> ... 8 more
> авг. 07, 2020 3:44:42 PM org.apache.ignite.logger.java.JavaLogger error
> SEVERE: Failed to unmarshal continuous routine handler [routineId=8f0137dc-cb32-40ff-83f4-867f272d338e, srcNodeId=ebcf4d5b-a476-43d5-b057-00b5694104e6]
> class org.apache.ignite.internal.IgniteDeploymentCheckedException: Failed to obtain deployment for class: Client1$$Lambda$513/0x00000008003b8840
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryDeployableObject.unmarshal(CacheContinuousQueryDeployableObject.java:94)
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.p2pUnmarshal(CacheContinuousQueryHandler.java:1308)
> at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.p2pUnmarshal(CacheContinuousQueryHandler.java:1283)
> at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$8.run(GridContinuousProcessor.java:693)
> at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7162)
> at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:826)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)