You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by user123 <ma...@db.com> on 2017/01/10 18:01:29 UTC
class org.apache.ignite.IgniteCheckedException: Shared memory
segment has been closed
Hi
I have an issue with one of my prod server.
My setup is 3 jvm using Ignite in partitioned mode.
The host is Red Hat server 6.8.
Brief : One of my component has thrown an IgniteCheckedException.
The log extract is follows
2017-01-09 01:58:16 ERROR GridJobWorker:495 - Error sending reply for job
[nodeId=f2d102cf-32e0-4d93-9a50-7bd660368282,
jobId=86ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask,
dep=LocalDeployment [super=GridDeployment [ts=1483915184938, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@70dea4e,
clsLdrId=e69ac308951-777b183c-1237-4336-a16d-27b1ac72a5bb, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=o.a.i.i.processors.cache.GridCacheAdapter$SizeTask,
sesId=66ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282,
startTime=1483927096248, endTime=9223372036854775807,
taskNodeId=f2d102cf-32e0-4d93-9a50-7bd660368282,
clsLdr=sun.misc.Launcher$AppClassLoader@70dea4e, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false,
subjId=f2d102cf-32e0-4d93-9a50-7bd660368282, mapFut=IgniteFuture
[orig=GridFutureAdapter [resFlag=0, res=null, startTime=1483927096257,
endTime=0, ignoreInterrupts=false, state=INIT]]],
jobId=86ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282], job=SizeJob []]
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=f2d102cf-32e0-4d93-9a50-7bd660368282, addrs=[10.240.117.147, 127.0.0.1],
sockAddrs=[primary-server.uk.com/10.240.117.147:47502,
/10.240.117.147:47502, /127.0.0.1:47502], discPort=47502, order=3,
intOrder=3, lastExchangeTime=1483915532830, loc=false,
ver=1.6.0#20160518-sha1:0b22c45b, isClient=false], topic=TOPIC_TASK,
msg=GridJobExecuteResponse [nodeId=777b183c-1237-4336-a16d-27b1ac72a5bb,
sesId=66ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282,
jobId=86ece408951-f2d102cf-32e0-4d93-9a50-7bd660368282, gridEx=null,
isCancelled=false], policy=2]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1290)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1354)
at
org.apache.ignite.internal.processors.job.GridJobWorker.finishJob(GridJobWorker.java:777)
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:561)
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1161)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1766)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
at
org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: TcpDiscoveryNode
[id=f2d102cf-32e0-4d93-9a50-7bd660368282, addrs=[10.240.117.147, 127.0.0.1],
sockAddrs=[primary-server.uk.com/10.240.117.147:47502,
/10.240.117.147:47502, /127.0.0.1:47502], discPort=47502, order=3,
intOrder=3, lastExchangeTime=1483915532830, loc=false,
ver=1.6.0#20160518-sha1:0b22c45b, isClient=false]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1993)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1933)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1285)
... 14 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to send
message to remote node: IpcSharedMemoryClientEndpoint
[inSpace=IpcSharedMemorySpace [opSize=262144, shmemPtr=140348055875648,
shmemId=1770946565, semId=1674903563, closed=false, isReader=true,
writerPid=25111, readerPid=24088,
tokFileName=/tmp/ignite/work/ipc/shmem/f2d102cf-32e0-4d93-9a50-7bd660368282-25111/gg-shmem-space-1-24088-262144,
closed=false], outSpace=IpcSharedMemorySpace [opSize=262144,
shmemPtr=140347921915968, shmemId=1770913796, semId=1674870793,
closed=false, isReader=false, writerPid=24088, readerPid=25111,
tokFileName=/tmp/ignite/work/ipc/shmem/f2d102cf-32e0-4d93-9a50-7bd660368282-25111/gg-shmem-space-0-24088-262144,
closed=false], checkIn=true, checkOut=true]
at
org.apache.ignite.internal.util.nio.GridShmemCommunicationClient.sendMessage(GridShmemCommunicationClient.java:137)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1974)
... 16 more
Caused by: java.io.IOException: class
org.apache.ignite.IgniteCheckedException: Shared memory segment has been
closed
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryOutputStream.write(IpcSharedMemoryOutputStream.java:56)
at
org.apache.ignite.internal.util.IgniteUtils.writeMessageFully(IgniteUtils.java:9388)
at
org.apache.ignite.internal.util.nio.GridShmemCommunicationClient.sendMessage(GridShmemCommunicationClient.java:132)
... 17 more
Caused by: class org.apache.ignite.IgniteCheckedException: Shared memory
segment has been closed
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryUtils.writeSharedMemory(Native
Method)
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemorySpace.write(IpcSharedMemorySpace.java:159)
at
org.apache.ignite.internal.util.ipc.shmem.IpcSharedMemoryOutputStream.write(IpcSharedMemoryOutputStream.java:53)
... 19 more
My Ignite is version 1.6.
My ignite config is
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean abstract="true" id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="metricsLogFrequency" value="0"/>
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Segment resources are
ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 32768
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536
ipcs -pm
------ Shared Memory Creator/Last-op --------
shmid owner cpid lpid
1934557184 xxxx 26166 24088
1934589953 xxxx 26166 24088
1934622722 xxxx 25111 24088
1934655491 xxxx 25111 24088
1771044872 xxxx 26166 25111
1771077641 xxxx 26166 25111
1771175946 xxxx 25111 26166
1771208715 xxxx 25111 26166
I would appreciate any inputs on the cause of this Segment Closed....
Regards
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/class-org-apache-ignite-IgniteCheckedException-Shared-memory-segment-has-been-closed-tp9996.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: class org.apache.ignite.IgniteCheckedException: Shared memory
segment has been closed
Posted by vkulichenko <va...@gmail.com>.
Hi,
Shared memory is known to be not very stable under load, and it will not be
default anymore in 2.0 (and will probably be deprecated). I would recommend
to disable it, just add the following into configuration:
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="sharedMemoryPort" value="-1"/>
</bean>
</property>
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/class-org-apache-ignite-IgniteCheckedException-Shared-memory-segment-has-been-closed-tp9996p10003.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.