You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by user123 <ma...@db.com> on 2017/01/10 17:48:34 UTC
ignitecheckedException thrown
Hi
I have a setup of 3 jvm all talking to each other using Ignite 1.6
one of my application(jvm) has an exception thrown by the Ignite framework,
which I am not able to understand.
The exception trace below is from one of the process.
I am using Red Hat Linux server 6.8.
During this exception, the other 2 processes had no activities (so far as I
can see from their logs).
My questions are
1) The root cause appears to be org.apache.ignite.IgniteCheckedException:
Shared memory segment has been closed...
2) Looking at the server in question
ipcs -l shows the following
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 32768
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536
and ipcs -pm shows
------ Shared Memory Creator/Last-op --------
shmid owner cpid lpid
1934557184 xxxx 26166 24088
1934589953 xxxx 26166 24088
1934622722 xxxx 25111 24088
1934655491 xxxx 25111 24088
1771044872 xxxx 26166 25111
1771077641 xxxx 26166 25111
1771175946 xxxx 25111 26166
1771208715 xxxx 25111 26166
I would appreciate for any inputs to the cause of my issue and also whether
this makes my affected component non longer stable.
The log trace output.
===================
2017-01-09 01:58:16 ERROR GridJobWorker:495 - Error sending reply for job
[nodeId=f2d102cf-32e0-4d93-9a50-7bd660368282,
jobId=86ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask,
dep=LocalDeployment [super=GridDeployment [ts=1483915184938, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@70dea4e,
clsLdrId=e69ac308951-777b183c-1237-4336-a16d-27b1ac72a5bb, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask,
sesId=66ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282,
startTime=1483927096248, endTime=9223372036854775807,
taskNodeId=f2d102cf-32e0-4d93-9a50-7bd660368282,
clsLdr=sun.misc.Launcher$AppClassLoader@70dea4e, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false,
subjId=f2d102cf-32e0-4d93-9a50-7bd660368282, mapFut=IgniteFuture
[orig=GridFutureAdapter [resFlag=0, res=null, startTime=1483927096257,
endTime=0, ignoreInterrupts=false, state=INIT]]],
jobId=86ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282], job=SizeJob []]
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=f2d102cf-32e0-4d93-9a50-7bd660368282, addrs=[10.240.117.147, 127.0.0.1],
sockAddrs=[gcctrp01.uk.db.com/10.240.117.147:47502, /10.240.117.147:47502,
/127.0.0.1:47502], discPort=47502, order=3, intOrder=3,
lastExchangeTime=1483915532830, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], topic=TOPIC_TASK, msg=GridJobExecuteResponse
[nodeId=777b183c-1237-4336-a16d-27b1ac72a5bb,
sesId=66ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282,
jobId=86ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282, gridEx=null,
isCancelled=false], policy=2]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1290)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1354)
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:777)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:561)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1161)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1766)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
at
org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=f2d102cf-32e0-4d93-9a50-7bd660368282, addrs=[10.240.117.147, 127.0.0.1],
sockAddrs=[gcctrp01.uk.db.com/10.240.117.147:47502, /10.240.117.147:47502,
/127.0.0.1:47502], discPort=47502, order=3, intOrder=3,
lastExchangeTime=1483915532830, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1993)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1933)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1285)
... 14 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to send
message to remote node: IpcSharedMemoryClientEndpoint
[inSpace=IpcSharedMemorySpace [opSize=262144, shmemPtr=140348055875648,
shmemId=1770946565, semId=1674903563, closed=false, isReader=true,
writerPid=25111, readerPid=24088,
tokFileName=/tmp/ignite/work/ipc/shmem/f2d102cf-32e0-4d93-9a50-7bd660368282-25111/gg-shmem-space-1-24088-262144,
closed=false], outSpace=IpcSharedMemorySpace [opSize=262144,
shmemPtr=140347921915968, shmemId=1770913796, semId=1674870793,
closed=false, isReader=false, writerPid=24088, readerPid=25111,
tokFileName=/tmp/ignite/work/ipc/shmem/f2d102cf-32e0-4d93-9a50-7bd660368282-25111/gg-shmem-space-0-24088-262144,
closed=false], checkIn=true, checkOut=true]
at
org.apache.ignite.internal.util.nio.GridShmemCommunicationClient.sendMessage(GridShmemCommunicationClient.java:137)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1974)
... 16 more
Caused by: java.io.IOException: class
org.apache.ignite.IgniteCheckedException: Shared memory segment has been
closed
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryOutputStream.write(IpcSharedMemoryOutputStream.java:56)
at
org.apache.ignite.internal.util.IgniteUtils.writeMessageFully(IgniteUtils.java:9388)
at
org.apache.ignite.internal.util.nio.GridShmemCommunicationClient.sendMessage(GridShmemCommunicationClient.java:132)
... 17 more
Caused by: class org.apache.ignite.IgniteCheckedException: Shared memory
segment has been closed
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryUtils.writeSharedMemory(Native
Method)
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemorySpace.write(IpcSharedMemorySpace.java:159)
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryOutputStream.write(IpcSharedMemoryOutputStream.java:53)
... 19 more
===================
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignitecheckedException-thrown-tp9995.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.