You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ranjit Sahu <ra...@gmail.com> on 2017/10/23 16:00:06 UTC
intermittent exceptions while starting ignite clsuter
Hi Team,
I am geting the attached exception once in a while while starting the
ignite cluster.
Even though no one is running ignite i get some times address already in
use.
I have specified a ramnge of 10 for both the discovery and communication
ports.
Any help what i am missing here.
my code below
val igniteConfig = new IgniteConfiguration()
igniteConfig.setGridName(igniteGridName)
val tcpDiscoverySpi = new TcpDiscoverySpi()
val ipFinder = new TcpDiscoveryVmIpFinder()
ipFinder.setAddresses(ipAddresses.value.toList.asJava)
tcpDiscoverySpi.setIpFinder(ipFinder)
tcpDiscoverySpi.setLocalPort(Integer.valueOf(portNumber.value))
tcpDiscoverySpi.setLocalPortRange(10)
igniteConfig.setDiscoverySpi(tcpDiscoverySpi)
val commSpi=new TcpCommunicationSpi()
commSpi.setLocalPort(Integer.valueOf(portNumber.value)+1000)
commSpi.setLocalPortRange(10)
// Overriding communication SPI.
igniteConfig.setCommunicationSpi(commSpi)
Ignition.getOrStart(igniteConfig)
Thanks,
Ranjit
17/10/23 10:46:06 WARN scheduler.TaskSetManager: Lost task 18.0 in
stage 1.0 (TID 58, c157mgh.int.westgroup.com, executor 25): class
org.apache.ignite.IgniteException: Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:927)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:417)
at un.api.resolver.dao.CompanyDAO.getIgnite(CompanyDAO.scala:92)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:71)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:67)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
start manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1589)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:857)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1794)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1598)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1041)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:568)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:529)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
get SPI attributes.
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:249)
at org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:231)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1584)
... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
initialize TCP server: 0.0.0.0/0.0.0.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1481)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:232)
... 22 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
bind to any port within range [startPort=48750, portRange=10,
locHost=0.0.0.0/0.0.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1762)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1478)
... 23 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize NIO selector.
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:670)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:289)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:88)
at org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:2439)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1711)
... 24 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:654)
... 28 more
17/10/23 10:46:06 INFO scheduler.TaskSetManager: Starting task 18.1 in
stage 1.0 (TID 80, xxxxx, executor 25, partition 18, PROCESS_LOCAL,
1965 bytes)
17/10/23 10:46:06 INFO scheduler.TaskSetManager: Lost task 39.0 in
stage 1.0 (TID 79) on xxxx, executor 25:
org.apache.ignite.IgniteException (Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager])
[duplicate 1]
17/10/23 10:46:06 INFO scheduler.TaskSetManager: Starting task 39.1 in
stage 1.0 (TID 81, xxxx, executor 25, partition 39, PROCESS_LOCAL,
1932 bytes)
17/10/23 10:46:06 INFO scheduler.TaskSetManager: Lost task 18.1 in
stage 1.0 (TID 80) on xxxxx, executor 25:
org.apache.ignite.IgniteException (Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager])
[duplicate 2]
17/10/23 10:46:06 ERROR scheduler.TaskSetManager: Task 18 in stage 1.0
failed 4 times; aborting job
17/10/23 10:46:06 INFO cluster.YarnClusterScheduler: Cancelling stage 1
17/10/23 10:46:06 INFO cluster.YarnClusterScheduler: Stage 1 was cancelled
17/10/23 10:46:06 INFO scheduler.DAGScheduler: ResultStage 1
(foreachPartition at IgniteDataLoader.scala:67) failed in 1.324 s due
to Job aborted due to stage failure: Task 18 in stage 1.0 failed 4
times, most recent failure: Lost task 18.3 in stage 1.0 (TID 84,
c157mgh.int.westgroup.com, executor 25): class
org.apache.ignite.IgniteException: Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:927)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:417)
at un.api.resolver.dao.CompanyDAO.getIgnite(CompanyDAO.scala:92)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:71)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:67)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
start manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1589)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:857)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1794)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1598)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1041)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:568)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:529)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
get SPI attributes.
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:249)
at org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:231)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1584)
... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
initialize TCP server: 0.0.0.0/0.0.0.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1481)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:232)
... 22 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
bind to any port within range [startPort=48750, portRange=10,
locHost=0.0.0.0/0.0.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1762)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1478)
... 23 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize NIO selector.
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:670)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:289)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:88)
at org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:2439)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1711)
... 24 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:654)
... 28 more
Driver stacktrace:
17/10/23 10:46:06 INFO scheduler.DAGScheduler: Job 1 failed:
foreachPartition at IgniteDataLoader.scala:67, took 1.346020 s
17/10/23 10:46:06 ERROR yarn.ApplicationMaster: User class threw
exception: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 18 in stage 1.0 failed 4 times, most recent failure:
Lost task 18.3 in stage 1.0 (TID 84, xxxxx, executor 25): class
org.apache.ignite.IgniteException: Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:927)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:417)
at un.api.resolver.dao.CompanyDAO.getIgnite(CompanyDAO.scala:92)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:71)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:67)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
start manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1589)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:857)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1794)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1598)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1041)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:568)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:529)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
get SPI attributes.
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:249)
at org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:231)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1584)
... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
initialize TCP server: 0.0.0.0/0.0.0.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1481)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:232)
... 22 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
bind to any port within range [startPort=48750, portRange=10,
locHost=0.0.0.0/0.0.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1762)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1478)
... 23 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize NIO selector.
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:670)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:289)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:88)
at org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:2439)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1711)
... 24 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:654)
... 28 more
Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure:
Task 18 in stage 1.0 failed 4 times, most recent failure: Lost task
18.3 in stage 1.0 (TID 84, c157mgh.int.westgroup.com, executor 25):
class org.apache.ignite.IgniteException: Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:927)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:417)
at un.api.resolver.dao.CompanyDAO.getIgnite(CompanyDAO.scala:92)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:71)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:67)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
start manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1589)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:857)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1794)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1598)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1041)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:568)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:529)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
get SPI attributes.
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:249)
at org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:231)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1584)
... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
initialize TCP server: 0.0.0.0/0.0.0.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1481)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:232)
... 22 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
bind to any port within range [startPort=48750, portRange=10,
locHost=0.0.0.0/0.0.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1762)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1478)
... 23 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize NIO selector.
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:670)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:289)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:88)
at org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:2439)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1711)
... 24 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:654)
... 28 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1840)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1853)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1866)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1937)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:918)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:918)
at un.api.dataloader.IgniteDataLoader$.initIgniteCluster(IgniteDataLoader.scala:67)
at un.api.resolver.accessor.UnifiedAccessor$.getCompanyAccessor(UnifiedAccessor.scala:110)
at un.api.resolver.CompanyResolverSparkTest$.main(CompanyResolverSparkTest.scala:33)
at un.api.resolver.CompanyResolverSparkTest.main(CompanyResolverSparkTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:553)
Caused by: class org.apache.ignite.IgniteException: Failed to start
manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:927)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:417)
at un.api.resolver.dao.CompanyDAO.getIgnite(CompanyDAO.scala:92)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:71)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:67)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
start manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1589)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:857)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1794)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1598)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1041)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:568)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:529)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
get SPI attributes.
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:249)
at org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:231)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1584)
... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
initialize TCP server: 0.0.0.0/0.0.0.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1481)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:232)
... 22 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
bind to any port within range [startPort=48750, portRange=10,
locHost=0.0.0.0/0.0.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1762)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1478)
... 23 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize NIO selector.
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:670)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:289)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:88)
at org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:2439)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1711)
... 24 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:654)
... 28 more
17/10/23 10:46:06 INFO yarn.ApplicationMaster: Final app status:
FAILED, exitCode: 15, (reason: User class threw exception:
org.apache.spark.SparkException: Job aborted due to stage failure:
Task 18 in stage 1.0 failed 4 times, most recent failure: Lost task
18.3 in stage 1.0 (TID 84, c157mgh.int.westgroup.com, executor 25):
class org.apache.ignite.IgniteException: Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:927)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:417)
at un.api.resolver.dao.CompanyDAO.getIgnite(CompanyDAO.scala:92)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:71)
at un.api.dataloader.IgniteDataLoader$$anonfun$initIgniteCluster$2.apply(IgniteDataLoader.scala:67)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1866)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
start manager: GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.communication.GridIoManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1589)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:857)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1794)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1598)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1041)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:568)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:529)
at org.apache.ignite.Ignition.getOrStart(Ignition.java:414)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
get SPI attributes.
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:249)
at org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:231)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1584)
... 20 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to
initialize TCP server: 0.0.0.0/0.0.0.0
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1481)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:232)
... 22 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
bind to any port within range [startPort=48750, portRange=10,
locHost=0.0.0.0/0.0.0.0]
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1762)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:1478)
... 23 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize NIO selector.
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:670)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:289)
at org.apache.ignite.internal.util.nio.GridNioServer.<init>(GridNioServer.java:88)
at org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:2439)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:1711)
... 24 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:654)
... 28 more
Re: intermittent exceptions while starting ignite clsuter
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi,
Let's start with communication spi first. Please configure communication
ports in that way and check that discovery ports and communication are not
intersected.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: intermittent exceptions while starting ignite clsuter
Posted by Ranjit Sahu <ra...@gmail.com>.
for both discovery as well as communication spi you mean ?
On Mon, Oct 23, 2017 at 11:26 PM, slava.koptilin <sl...@gmail.com>
wrote:
> Hi Ranjit,
>
> It seems that all ports in the range you provided are already bound.
> Perhaps, your machine doesn't release ports fast enough.
> Please try to increase the port range up to 100.
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: intermittent exceptions while starting ignite clsuter
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Ranjit,
It seems that all ports in the range you provided are already bound.
Perhaps, your machine doesn't release ports fast enough.
Please try to increase the port range up to 100.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/