You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ivan Bessonov (Jira)" <ji...@apache.org> on 2020/06/16 14:16:00 UTC
[jira] [Updated] (IGNITE-13156) Continuous query filter deployment
hungs discovery thread
[ https://issues.apache.org/jira/browse/IGNITE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Bessonov updated IGNITE-13156:
-----------------------------------
Description:
Continuous query starts with a custom discovery event. Handler of the event is executed in discovery thread synchronously. Even worse is the fact that message itself is mutable and it blocks the ring.
Inside of the handler there is a is p2p resource request from other node, which can be pretty time consuming. And after https://issues.apache.org/jira/browse/IGNITE-12438 or similar tasks this could even lead to a deadlock.
All IO operations must be removed from discovery handlers.
This scenario is reproduced in GridP2PContinuousDeploymentSelfTest#testServerJoinWithP2PClassDeployedInCluster
{code:java}
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2231) at org.apache.ignite.internal.managers.deployment.GridDeploymentCommunication.sendResourceRequest(GridDeploymentCommunication.java:456) at org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendResourceRequest(GridDeploymentClassLoader.java:793) at org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.getResourceAsStreamEx(GridDeploymentClassLoader.java:745) at org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.checkLoadRemoteClass(GridDeploymentPerVersionStore.java:729) at org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.getDeployment(GridDeploymentPerVersionStore.java:314) at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getGlobalDeployment(GridDeploymentManager.java:498) at org.apache.ignite.internal.GridEventConsumeHandler.p2pUnmarshal(GridEventConsumeHandler.java:416) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1423) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:670) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:533) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2635) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2673) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748)
{code}
was:
Continuous query starts with a custom discovery event. Handler of the event is executed in discovery thread synchronously. Even worse is the fact that message itself is mutable and it blocks the ring.
Inside of the handler there is a is p2p resource request from other node, which can be pretty time consuming. And after https://issues.apache.org/jira/browse/IGNITE-12438 or similar tasks this could even lead to a deadlock.
All IO operations must be removed from discovery handlers.
{code:java}
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2231) at org.apache.ignite.internal.managers.deployment.GridDeploymentCommunication.sendResourceRequest(GridDeploymentCommunication.java:456) at org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendResourceRequest(GridDeploymentClassLoader.java:793) at org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.getResourceAsStreamEx(GridDeploymentClassLoader.java:745) at org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.checkLoadRemoteClass(GridDeploymentPerVersionStore.java:729) at org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.getDeployment(GridDeploymentPerVersionStore.java:314) at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getGlobalDeployment(GridDeploymentManager.java:498) at org.apache.ignite.internal.GridEventConsumeHandler.p2pUnmarshal(GridEventConsumeHandler.java:416) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1423) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:670) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:533) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2635) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2673) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748)
{code}
> Continuous query filter deployment hungs discovery thread
> ---------------------------------------------------------
>
> Key: IGNITE-13156
> URL: https://issues.apache.org/jira/browse/IGNITE-13156
> Project: Ignite
> Issue Type: Bug
> Reporter: Ivan Bessonov
> Priority: Major
>
> Continuous query starts with a custom discovery event. Handler of the event is executed in discovery thread synchronously. Even worse is the fact that message itself is mutable and it blocks the ring.
> Inside of the handler there is a is p2p resource request from other node, which can be pretty time consuming. And after https://issues.apache.org/jira/browse/IGNITE-12438 or similar tasks this could even lead to a deadlock.
> All IO operations must be removed from discovery handlers.
> This scenario is reproduced in GridP2PContinuousDeploymentSelfTest#testServerJoinWithP2PClassDeployedInCluster
> {code:java}
> at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2231) at org.apache.ignite.internal.managers.deployment.GridDeploymentCommunication.sendResourceRequest(GridDeploymentCommunication.java:456) at org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendResourceRequest(GridDeploymentClassLoader.java:793) at org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.getResourceAsStreamEx(GridDeploymentClassLoader.java:745) at org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.checkLoadRemoteClass(GridDeploymentPerVersionStore.java:729) at org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.getDeployment(GridDeploymentPerVersionStore.java:314) at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getGlobalDeployment(GridDeploymentManager.java:498) at org.apache.ignite.internal.GridEventConsumeHandler.p2pUnmarshal(GridEventConsumeHandler.java:416) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1423) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:670) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:533) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2635) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2673) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)