You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by baozipu <jo...@gmail.com> on 2017/05/08 21:19:00 UTC
How to correctly shut down Ignite Application
Hello, here is my topology configuration:
1. One Ignite server application, which holds some data in the RDD.
2. One ignite client application, which retrieves the data and do some
processing.
There is no problem on the server side, until the client side has done the
job and exit, by calling:
igniteContext.close()
sparkSession.close()
The server side got these error messages:
17/05/08 13:46:49 WARN TcpDiscoverySpi: Received EVT_NODE_FAILED event with
warning [nodeInitiatedEvt=TcpDiscoveryNode
[id=7c720694-b34b-4a83-a131-37f3b7dc95b5, addrs=[0:0:0:0:0:0:0:1%1,
10.93.128.11, 10.93.136.11, 127.0.0.1],
sockAddrs=[mapr271-eth1.sv.mynode.com/10.93.136.11:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500, mapr271.sv.mynode.com/10.93.128.11:47500],
discPort=47500, order=17, intOrder=17, lastExchangeTime=1494276394774,
loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
msg=TcpCommunicationSpi failed to establish connection to node
[rmtNode=TcpDiscoveryNode [id=2b404f04-cad5-4209-a615-ab3b9639b8e4,
addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.89],
sockAddrs=[sv1-hp0204-08.sv.mynode.com/172.30.32.89:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=13, intOrder=13,
lastExchangeTime=1494276395700, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], errs=class o.a.i.IgniteCheckedException: Failed to connect
to node (is node still alive?). Make sure that each ComputeTask and cache
Transaction has a timeout set in order to prevent parties from waiting
forever in case of network issues
[nodeId=2b404f04-cad5-4209-a615-ab3b9639b8e4,
addrs=[sv1-hp0204-08.sv.mynode.com/172.30.32.89:47100, /127.0.0.1:47100,
/0:0:0:0:0:0:0:1%1:47100]], connectErrs=[class o.a.i.IgniteCheckedException:
Failed to connect to address:
sv1-hp0204-08.sv.mynode.com/172.30.32.89:47100, class
o.a.i.IgniteCheckedException: Failed to connect to address:
/127.0.0.1:47100, class o.a.i.IgniteCheckedException: Failed to connect to
address: /0:0:0:0:0:0:0:1%1:47100]]]
17/05/08 13:46:49 WARN GridDiscoveryManager: Node FAILED: TcpDiscoveryNode
[id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.32.89],
sockAddrs=[sv1-hp0204-08.mynode.com/172.30.32.89:47500, /127.0.0.1:47500,
/0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=13, intOrder=13,
lastExchangeTime=1494276390779, loc=false, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false]
Where is the problem, how can I solve it? Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-Application-tp12548.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
RE: How to correctly shut down Ignite Application
Posted by Juan Barani <Ju...@unitedplanet.com>.
Thanks Andrey,
For the moment it works correctly with preferIPv4Stack. The issue first appeared on 2.0, we run the same test suite since 1.6, and we have never seen it.
Thanks again,
Juan
From: Andrey Mashenkov [mailto:andrey.mashenkov@gmail.com]
Sent: Wednesday, May 10, 2017 12:32 PM
To: user@ignite.apache.org
Subject: Re: How to correctly shut down Ignite Application
Hi Juan,
Looks like there is a bug, sometimes ignite choose ipv6 protocol to communicate with other nodes that causes strange communication errors.
On Wed, May 10, 2017 at 1:01 PM, Juan Barani <Ju...@unitedplanet.com>> wrote:
Hi,
Not necessarily is the case that the client cannot connect to the server. I experienced yesterday with ignite 2.0, that when a client is forced with:
l_discoverySpi.setForceServerMode(true);
it comes as isClient=false in the exception message, even if setClientMode=true.
Since updating to 2.0, we experienced random NPEs when publishing a message in a topic during the interval a server went down, but failed to notify other nodes due to a hard stop =>
ERROR 2017-05-10T09:57:58,282 - de.uplanet.test.integration.RemoteTestServiceBean[pool-4-thread-1]
Failed to send message to remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0<http://172.17.0.1:0>, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0<http://127.0.0.1:0>, /10.0.2.15:0<http://10.0.2.15:0>], discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true]
org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0<http://172.17.0.1:0>, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0<http://127.0.0.1:0>, /10.0.2.15:0<http://10.0.2.15:0>], discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2483) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2419) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1329) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1698) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessageToGridTopic(GridIoManager.java:1473) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.sendUserMessage(GridIoManager.java:1588) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.IgniteMessagingImpl.sendOrdered(IgniteMessagingImpl.java:165) ~[ignite-core-2.0.0.jar:2.0.0]
at de.uplanet.lucy.server.distributed.cloud.datagrid.ignite.IgniteGridTopic.publish(IgniteGridTopic.java:58) ~[update/:?]
at de.uplanet.test.integration.RemoteTestServiceBean.lambda$3(RemoteTestServiceBean.java:123) ~[update/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_92]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_92]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_92]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_92]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
Caused by: org.apache.ignite.IgniteCheckedException: java.lang.NullPointerException
at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7242) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2630) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455) ~[ignite-core-2.0.0.jar:2.0.0]
... 13 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_92]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_92]
at org.apache.ignite.internal.util.IgniteUtils.filterReachable(IgniteUtils.java:1895) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2891) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2702) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2594) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455) ~[ignite-core-2.0.0.jar:2.0.0]
... 13 more
Caused by: java.lang.NullPointerException
at org.apache.ignite.internal.util.IgniteUtils.reachable(IgniteUtils.java:2102) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.IgniteUtils$18.run(IgniteUtils.java:1884) ~[ignite-core-2.0.0.jar:2.0.0]
... 5 more
These errors disappeared after setting: -Djava.net.preferIPv4Stack=true
I came under the impression that something is working different under ipv4 and ipv6, but cannot tell why the NPE was coming.
Probably the errors are related.
Thanks!
Juan
From: Andrey Mashenkov [mailto:andrey.mashenkov@gmail.com<ma...@gmail.com>]
Sent: Tuesday, May 9, 2017 7:49 PM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: How to correctly shut down Ignite Application
Looks like client node can't connect to server
[id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.32.89].. isClient=false]
Would you please attach full logs?
On Tue, May 9, 2017 at 2:55 AM, baozipu <jo...@gmail.com>> wrote:
Hi,
Thanks for the reply. Both the applications are launched in Embedded mode in
Spark. I am not sure if the flag of preferIPv4Stack still matters.
Both of the applications runs perfect when they are alive. It is only when
the "client application" (actually another application in server mode)
closed, exit from the topology, that those error messages pop up.
I couldn't understand why the server application reports such error. It
should be normal that some connections disconnected....
Could you give me more detail for diagnosing the problem?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-Application-tp12548p12553.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
--
Best regards,
Andrey V. Mashenkov
--
Best regards,
Andrey V. Mashenkov
Re: How to correctly shut down Ignite Application
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Juan,
Looks like there is a bug, sometimes ignite choose ipv6 protocol to
communicate with other nodes that causes strange communication errors.
On Wed, May 10, 2017 at 1:01 PM, Juan Barani <Ju...@unitedplanet.com>
wrote:
> Hi,
>
>
>
> Not necessarily is the case that the client cannot connect to the server.
> I experienced yesterday with ignite 2.0, that when a client is forced with:
>
>
>
> l_discoverySpi.setForceServerMode(true);
>
>
>
> it comes as isClient=false in the exception message, even if
> setClientMode=true.
>
>
>
> Since updating to 2.0, we experienced random NPEs when publishing a
> message in a topic during the interval a server went down, but failed to
> notify other nodes due to a hard stop =>
>
>
>
> ERROR 2017-05-10T09:57:58,282 - de.uplanet.test.integration.
> RemoteTestServiceBean[pool-4-thread-1]
>
> Failed to send message to remote node: TcpDiscoveryNode
> [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo,
> 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0,
> 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /10.0.2.15:0], discPort=0, order=3,
> intOrder=3, lastExchangeTime=1494410235152, loc=false,
> ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true]
>
> org.apache.ignite.spi.IgniteSpiException: Failed to send message to
> remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd,
> addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/
> 172.17.0.1:0, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /10.0.2.15:0],
> discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152,
> loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2483)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2419)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.send(GridIoManager.java:1329) ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.send(GridIoManager.java:1698) ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.sendOrderedMessageToGridTopic(GridIoManager.java:1473)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.sendUserMessage(GridIoManager.java:1588)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.IgniteMessagingImpl.
> sendOrdered(IgniteMessagingImpl.java:165) ~[ignite-core-2.0.0.jar:2.0.0]
>
> at de.uplanet.lucy.server.distributed.cloud.datagrid.
> ignite.IgniteGridTopic.publish(IgniteGridTopic.java:58) ~[update/:?]
>
> at de.uplanet.test.integration.
> RemoteTestServiceBean.lambda$3(RemoteTestServiceBean.java:123)
> ~[update/:?]
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_92]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_92]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_92]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_92]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
>
> Caused by: org.apache.ignite.IgniteCheckedException:
> java.lang.NullPointerException
>
> at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7242)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.
> util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.
> util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.
> util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2630)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> ... 13 more
>
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[?:1.8.0_92]
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[?:1.8.0_92]
>
> at org.apache.ignite.internal.util.IgniteUtils.
> filterReachable(IgniteUtils.java:1895) ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2891)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2702)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2594)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> ... 13 more
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.ignite.internal.util.IgniteUtils.reachable(IgniteUtils.java:2102)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> at org.apache.ignite.internal.util.IgniteUtils$18.run(IgniteUtils.java:1884)
> ~[ignite-core-2.0.0.jar:2.0.0]
>
> ... 5 more
>
>
>
> These errors disappeared after setting: -Djava.net.preferIPv4Stack=true
>
>
>
> I came under the impression that something is working different under ipv4
> and ipv6, but cannot tell why the NPE was coming.
>
>
>
> Probably the errors are related.
>
>
>
> Thanks!
>
>
>
> Juan
>
>
>
> *From:* Andrey Mashenkov [mailto:andrey.mashenkov@gmail.com]
> *Sent:* Tuesday, May 9, 2017 7:49 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: How to correctly shut down Ignite Application
>
>
>
> Looks like client node can't connect to server
>
> [id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1,
> 127.0.0.1, 172.30.32.89].. *isClient=false*]
>
>
>
> Would you please attach full logs?
>
>
>
> On Tue, May 9, 2017 at 2:55 AM, baozipu <jo...@gmail.com> wrote:
>
> Hi,
>
> Thanks for the reply. Both the applications are launched in Embedded mode
> in
> Spark. I am not sure if the flag of preferIPv4Stack still matters.
>
> Both of the applications runs perfect when they are alive. It is only when
> the "client application" (actually another application in server mode)
> closed, exit from the topology, that those error messages pop up.
>
> I couldn't understand why the server application reports such error. It
> should be normal that some connections disconnected....
>
> Could you give me more detail for diagnosing the problem?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-
> Application-tp12548p12553.html
>
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
>
>
> --
>
> Best regards,
> Andrey V. Mashenkov
>
--
Best regards,
Andrey V. Mashenkov
RE: How to correctly shut down Ignite Application
Posted by Juan Barani <Ju...@unitedplanet.com>.
Hi,
Not necessarily is the case that the client cannot connect to the server. I experienced yesterday with ignite 2.0, that when a client is forced with:
l_discoverySpi.setForceServerMode(true);
it comes as isClient=false in the exception message, even if setClientMode=true.
Since updating to 2.0, we experienced random NPEs when publishing a message in a topic during the interval a server went down, but failed to notify other nodes due to a hard stop =>
ERROR 2017-05-10T09:57:58,282 - de.uplanet.test.integration.RemoteTestServiceBean[pool-4-thread-1]
Failed to send message to remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /10.0.2.15:0], discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true]
org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /10.0.2.15:0], discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2483) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2419) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1329) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1698) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessageToGridTopic(GridIoManager.java:1473) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.managers.communication.GridIoManager.sendUserMessage(GridIoManager.java:1588) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.IgniteMessagingImpl.sendOrdered(IgniteMessagingImpl.java:165) ~[ignite-core-2.0.0.jar:2.0.0]
at de.uplanet.lucy.server.distributed.cloud.datagrid.ignite.IgniteGridTopic.publish(IgniteGridTopic.java:58) ~[update/:?]
at de.uplanet.test.integration.RemoteTestServiceBean.lambda$3(RemoteTestServiceBean.java:123) ~[update/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_92]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_92]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_92]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_92]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
Caused by: org.apache.ignite.IgniteCheckedException: java.lang.NullPointerException
at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7242) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2630) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455) ~[ignite-core-2.0.0.jar:2.0.0]
... 13 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_92]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_92]
at org.apache.ignite.internal.util.IgniteUtils.filterReachable(IgniteUtils.java:1895) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2891) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2702) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2594) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455) ~[ignite-core-2.0.0.jar:2.0.0]
... 13 more
Caused by: java.lang.NullPointerException
at org.apache.ignite.internal.util.IgniteUtils.reachable(IgniteUtils.java:2102) ~[ignite-core-2.0.0.jar:2.0.0]
at org.apache.ignite.internal.util.IgniteUtils$18.run(IgniteUtils.java:1884) ~[ignite-core-2.0.0.jar:2.0.0]
... 5 more
These errors disappeared after setting: -Djava.net.preferIPv4Stack=true
I came under the impression that something is working different under ipv4 and ipv6, but cannot tell why the NPE was coming.
Probably the errors are related.
Thanks!
Juan
From: Andrey Mashenkov [mailto:andrey.mashenkov@gmail.com]
Sent: Tuesday, May 9, 2017 7:49 PM
To: user@ignite.apache.org
Subject: Re: How to correctly shut down Ignite Application
Looks like client node can't connect to server
[id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.32.89].. isClient=false]
Would you please attach full logs?
On Tue, May 9, 2017 at 2:55 AM, baozipu <jo...@gmail.com>> wrote:
Hi,
Thanks for the reply. Both the applications are launched in Embedded mode in
Spark. I am not sure if the flag of preferIPv4Stack still matters.
Both of the applications runs perfect when they are alive. It is only when
the "client application" (actually another application in server mode)
closed, exit from the topology, that those error messages pop up.
I couldn't understand why the server application reports such error. It
should be normal that some connections disconnected....
Could you give me more detail for diagnosing the problem?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-Application-tp12548p12553.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
--
Best regards,
Andrey V. Mashenkov
Re: How to correctly shut down Ignite Application
Posted by Andrey Mashenkov <an...@gmail.com>.
Looks like client node can't connect to server
[id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1,
127.0.0.1, 172.30.32.89].. *isClient=false*]
Would you please attach full logs?
On Tue, May 9, 2017 at 2:55 AM, baozipu <jo...@gmail.com> wrote:
> Hi,
>
> Thanks for the reply. Both the applications are launched in Embedded mode
> in
> Spark. I am not sure if the flag of preferIPv4Stack still matters.
>
> Both of the applications runs perfect when they are alive. It is only when
> the "client application" (actually another application in server mode)
> closed, exit from the topology, that those error messages pop up.
>
> I couldn't understand why the server application reports such error. It
> should be normal that some connections disconnected....
>
> Could you give me more detail for diagnosing the problem?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-
> Application-tp12548p12553.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: How to correctly shut down Ignite Application
Posted by baozipu <jo...@gmail.com>.
Hi,
Thanks for the reply. Both the applications are launched in Embedded mode in
Spark. I am not sure if the flag of preferIPv4Stack still matters.
Both of the applications runs perfect when they are alive. It is only when
the "client application" (actually another application in server mode)
closed, exit from the topology, that those error messages pop up.
I couldn't understand why the server application reports such error. It
should be normal that some connections disconnected....
Could you give me more detail for diagnosing the problem?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-Application-tp12548p12553.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: How to correctly shut down Ignite Application
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Try to set same option for all nodes either -Djava.net.preferIPv4Stack=true
or -Djava.net.preferIPv6Stack=true.
There is an example in ignite.sh.
On Tue, May 9, 2017 at 12:19 AM, baozipu <jo...@gmail.com> wrote:
> Hello, here is my topology configuration:
>
> 1. One Ignite server application, which holds some data in the RDD.
> 2. One ignite client application, which retrieves the data and do some
> processing.
>
> There is no problem on the server side, until the client side has done the
> job and exit, by calling:
>
> igniteContext.close()
> sparkSession.close()
>
>
> The server side got these error messages:
>
> 17/05/08 13:46:49 WARN TcpDiscoverySpi: Received EVT_NODE_FAILED event with
> warning [nodeInitiatedEvt=TcpDiscoveryNode
> [id=7c720694-b34b-4a83-a131-37f3b7dc95b5, addrs=[0:0:0:0:0:0:0:1%1,
> 10.93.128.11, 10.93.136.11, 127.0.0.1],
> sockAddrs=[mapr271-eth1.sv.mynode.com/10.93.136.11:47500, /127.0.0.1:47500
> ,
> /0:0:0:0:0:0:0:1%1:47500, mapr271.sv.mynode.com/10.93.128.11:47500],
> discPort=47500, order=17, intOrder=17, lastExchangeTime=1494276394774,
> loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false],
> msg=TcpCommunicationSpi failed to establish connection to node
> [rmtNode=TcpDiscoveryNode [id=2b404f04-cad5-4209-a615-ab3b9639b8e4,
> addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.89],
> sockAddrs=[sv1-hp0204-08.sv.mynode.com/172.30.32.89:47500, /
> 127.0.0.1:47500,
> /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=13, intOrder=13,
> lastExchangeTime=1494276395700, loc=false, ver=1.9.0#20170302-sha1:
> a8169d0a,
> isClient=false], errs=class o.a.i.IgniteCheckedException: Failed to connect
> to node (is node still alive?). Make sure that each ComputeTask and cache
> Transaction has a timeout set in order to prevent parties from waiting
> forever in case of network issues
> [nodeId=2b404f04-cad5-4209-a615-ab3b9639b8e4,
> addrs=[sv1-hp0204-08.sv.mynode.com/172.30.32.89:47100, /127.0.0.1:47100,
> /0:0:0:0:0:0:0:1%1:47100]], connectErrs=[class
> o.a.i.IgniteCheckedException:
> Failed to connect to address:
> sv1-hp0204-08.sv.mynode.com/172.30.32.89:47100, class
> o.a.i.IgniteCheckedException: Failed to connect to address:
> /127.0.0.1:47100, class o.a.i.IgniteCheckedException: Failed to connect to
> address: /0:0:0:0:0:0:0:1%1:47100]]]
>
> 17/05/08 13:46:49 WARN GridDiscoveryManager: Node FAILED: TcpDiscoveryNode
> [id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1,
> 127.0.0.1, 172.30.32.89],
> sockAddrs=[sv1-hp0204-08.mynode.com/172.30.32.89:47500, /127.0.0.1:47500,
> /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=13, intOrder=13,
> lastExchangeTime=1494276390779, loc=false, ver=1.9.0#20170302-sha1:
> a8169d0a,
> isClient=false]
>
>
> Where is the problem, how can I solve it? Thanks.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-
> Application-tp12548.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov