You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by 波板仔 <37...@qq.com> on 2016/08/12 02:08:54 UTC
IgniteDataStreamer can't wirte data to ignite cluster
My ignite cluster is deploy on 2 machines. Each machine has 4 ignite process. like:
$jps
30678 CommandLineStartup
30723 CommandLineStartup
52461 Jps
30698 CommandLineStartup
30681 CommandLineStartup
When there is little data in the ignite cluster, I can use IgniteDataStreamer to write data into the ignite cluster. But if there have been more than 200G data in the ignite cluster, I can't use IgniteDataStreamer to write more data into the cluster. The memory and disk space are both enough. and the CPU load average is 2.09. The client log is :
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] Daemon mode: off
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS: Linux
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS user: mqq
[17:04:00,408][INFO][main][IgniteKernal%grid_recmd] Language runtime: Java Platform API Specification ver. 1.7
[17:04:00,408][INFO][main][IgniteKernal%grid_recmd] VM information: Java(TM) SE Runtime Environment 1.7.0_72-b14 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.72-b04
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] VM total memory: 2.9GB
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] Remote Management [restart: off, REST: on, JMX (remote: off)]
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] IGNITE_HOME=/data1/home/igni1.5
[17:04:00,410][INFO][main][IgniteKernal%grid_recmd] VM arguments: [-Xms3g, -Xmx3g, -XX:-UseGCOverheadLimit, -XX:+AggressiveOpts, -XX:MaxPermSize=256m, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+UseTLAB, -DIGNITE_QUIET=false]
[17:04:00,418][INFO][main][IgniteKernal%grid_recmd] Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache', 'cache_recmd']
[17:04:00,422][INFO][main][IgniteKernal%grid_recmd] 3-rd party licenses can be found at: /data1/home/igni1.5/libs/licenses
[17:04:00,518][INFO][main][IgniteKernal%grid_recmd] Non-loopback local IPs: 10.0.0.1
[17:04:00,519][INFO][main][IgniteKernal%grid_recmd] Enabled local MACs: 9017ACC1492D
[17:04:00,530][INFO][main][IgnitePluginProcessor] Configured plugins:
[17:04:00,530][INFO][main][IgnitePluginProcessor] ^-- None
[17:04:00,530][INFO][main][IgnitePluginProcessor]
[17:04:00,574][WARNING][main][TcpCommunicationSpi] Failure detection timeout will be ignored (one of SPI parameters has been set explicitly)
[17:04:00,627][INFO][main][TcpCommunicationSpi] IPC shared memory server endpoint started [port=48104, tokDir=/data1/home/igni1.5/work/ipc/shmem/e052dc07-8992-4834-87fd-abf84c0f644a-14472]
[17:04:00,628][INFO][main][TcpCommunicationSpi] Successfully bound shared memory communication to TCP port [port=48104, locHost=0.0.0.0/0.0.0.0]
[17:04:00,652][INFO][main][TcpCommunicationSpi] Successfully bound to TCP port [port=47106, locHost=0.0.0.0/0.0.0.0]
[17:04:00,685][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[17:04:00,749][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[17:04:00,755][WARNING][main][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi.
[17:04:00,757][INFO][main][IgniteKernal%grid_recmd] Security status [authentication=off, tls/ssl=off]
[17:04:00,861][WARNING][main][GridCacheProcessor] Deployment mode for cache is not CONTINUOUS or SHARED (it is recommended that you change deployment mode and restart): ISOLATED
[17:04:01,040][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11217]
[17:04:01,138][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,139][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,140][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,141][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,179][SEVERE][main][IgniteKernal%grid_recmd] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,180][SEVERE][main][IgniteKernal%grid_recmd] Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1510)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
... 12 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,193][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[17:05:01,201][WARNING][grid-nio-worker-0-#58%grid_recmd%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true]
[17:05:01,201][WARNING][grid-nio-worker-1-#59%grid_recmd%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true]
[17:05:01,843][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] Failed to read message [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a, rmtNodeId=198bc1a5-1b1d-4b65-bccb-b2eb5ad1ec9b]
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@3415ddf5
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:105)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:888)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.io.IOException: Stream closed.
at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:458)
at java.net.SocketInputStream.available(SocketInputStream.java:245)
at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3063)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:4890)
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.ArrayList.readObject(ArrayList.java:771)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
at java.util.TreeMap.readObject(TreeMap.java:3234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:102)
... 2 more
[17:05:01,845][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] Connection failed [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a]
java.io.IOException: Stream closed.
at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:458)
at java.net.SocketInputStream.available(SocketInputStream.java:245)
at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3063)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:4890)
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.ArrayList.readObject(ArrayList.java:771)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
at java.util.TreeMap.readObject(TreeMap.java:3234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:102)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:888)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,876][INFO][main][IgniteKernal%grid_recmd]
>>> +---------------------------------------------------------------------------------------+
>>> Ignite ver. 1.5.0-final#20151229-sha1:f1f8cda2f3f62231f42a59951bf34c39577c1bec stopped OK
>>> +---------------------------------------------------------------------------------------+
>>> Grid name: grid_recmd
>>> Grid uptime: 00:01:02:580
class org.apache.ignite.IgniteException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:882)
at org.apache.ignite.Ignition.start(Ignition.java:350)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1510)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
... 2 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
... 12 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
the ignite process's JVM_OPTS is :
export JVM_OPTS="-Xms5g -Xmx5g -Xmn2g -server -XX:-UseGCOverheadLimit -XX:+AggressiveOpts -XX:MaxPermSize=256m -XX:+UseP
arNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB"
the srv.xml is
<bean id="cacheA" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="cache_recmd"/>
<property name="cacheMode" value="PARTITIONED"/><!-- partition -->
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="hashIdResolver" >
<bean class="org.apache.ignite.cache.affinity.AffinityNodeAddressHashResolver"/>
</property>
<property name="partitions" value="280" />
</bean>
</property>
<property name="backups" value="0"/>
<property name="writeSynchronizationMode" value="PRIMARY_SYNC"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="memoryMode" value="ONHEAP_TIERED"/><!-- memory tire -->
<property name="startSize" value="#{30 * 1024 * 1024}"/>
<property name="offHeapMaxMemory" value="#{512 * 1024 * 1024}"/>
<property name="swapEnabled" value="true"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxMemorySize" value="#{1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
<property name="rebalanceThrottle" value="100"/>
</bean>
<bean id="igniA" class="org.apache.ignite.configuration.IgniteConfiguration" scope="singleton">
<property name="gridName" value="grid_recmd"/>
<property name="clientMode" value="false"/>
<property name="metricsLogFrequency" value="0"/><!-- 0 means disable metrics log -->
<property name="deploymentMode" value="ISOLATED"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="47500"/>
<property name="localPortRange" value="20"/>
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="shared" value="true"/>
<property name="addresses">
<list>
<value>10.0.0.1:47500..47520</value>
<value>10.0.0.2:47500..47520</value>
</list>
</property>
</bean>
</property>
<property name="joinTimeout" value="60000"/>
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="socketWriteTimeout" value="30000"/>
<property name="connectTimeout" value="50000"/>
</bean>
</property>
<property name="failureDetectionTimeout" value="60000"/>
<property name="swapSpaceSpi">
<bean class="org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi">
<property name="baseDirectory" value="/data1/home/igni1.5/my/swp"/>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<ref bean="cacheA"/>
</list>
</property>
<property name="includeEventTypes">
<list>
</list>
</property>
</bean>
Re: IgniteDataStreamer can't wirte data to ignite cluster
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hello,
The issue with grid segmented: The reasons of this may be long garbage
collector pause, long pause of process of user layer due to process of
system kernal work, network problem (like break).
By steps:
1) Update ignite version, because the 1.5.0 is very old. 1.7.0 is already
available.
2) Collect logs work garbage collector. You can use the article [1].
3) Check your system, using utils top, dstat, ping
It the issue continue to repeat, provide logs from server and client,
garbage collector logs and output the utils at the time of the crash.
On Fri, Aug 12, 2016 at 5:08 AM, 波板仔 <37...@qq.com> wrote:
> My ignite cluster is deploy on 2 machines. Each machine has 4 ignite
> process. like:
>
> $jps
> 30678 CommandLineStartup
> 30723 CommandLineStartup
> 52461 Jps
> 30698 CommandLineStartup
> 30681 CommandLineStartup
>
> When there is little data in the ignite cluster, I can use
> IgniteDataStreamer to write data into the ignite cluster. But if there have
> been more than 200G data in the ignite cluster, I can't use
> IgniteDataStreamer to write more data into the cluster. The memory and disk
> space are both enough. and the CPU load average is 2.09. The client log is :
>
>
> [17:04:00,407][INFO][main][IgniteKernal%grid_recmd] Daemon mode: off
> [17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS: Linux
> [17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS user: mqq
> [17:04:00,408][INFO][main][IgniteKernal%grid_recmd] Language runtime:
> Java Platform API Specification ver. 1.7
> [17:04:00,408][INFO][main][IgniteKernal%grid_recmd] VM information:
> Java(TM) SE Runtime Environment 1.7.0_72-b14 Oracle Corporation Java
> HotSpot(TM) 64-Bit Server VM 24.72-b04
> [17:04:00,409][INFO][main][IgniteKernal%grid_recmd] VM total memory:
> 2.9GB
> [17:04:00,409][INFO][main][IgniteKernal%grid_recmd] Remote Management
> [restart: off, REST: on, JMX (remote: off)]
> [17:04:00,409][INFO][main][IgniteKernal%grid_recmd]
> IGNITE_HOME=/data1/home/igni1.5
> [17:04:00,410][INFO][main][IgniteKernal%grid_recmd] VM arguments:
> [-Xms3g, -Xmx3g, -XX:-UseGCOverheadLimit, -XX:+AggressiveOpts,
> -XX:MaxPermSize=256m, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC,
> -XX:+UseTLAB, -DIGNITE_QUIET=false]
> [17:04:00,418][INFO][main][IgniteKernal%grid_recmd] Configured caches
> ['ignite-marshaller-sys-cache', 'ignite-sys-cache',
> 'ignite-atomics-sys-cache', 'cache_recmd']
> [17:04:00,422][INFO][main][IgniteKernal%grid_recmd] 3-rd party
> licenses can be found at: /data1/home/igni1.5/libs/licenses
> [17:04:00,518][INFO][main][IgniteKernal%grid_recmd] Non-loopback
> local IPs: 10.0.0.1
> [17:04:00,519][INFO][main][IgniteKernal%grid_recmd] Enabled local
> MACs: 9017ACC1492D
> [17:04:00,530][INFO][main][IgnitePluginProcessor] Configured plugins:
> [17:04:00,530][INFO][main][IgnitePluginProcessor] ^-- None
> [17:04:00,530][INFO][main][IgnitePluginProcessor]
> [17:04:00,574][WARNING][main][TcpCommunicationSpi] Failure detection
> timeout will be ignored (one of SPI parameters has been set explicitly)
> [17:04:00,627][INFO][main][TcpCommunicationSpi] IPC shared memory
> server endpoint started [port=48104, tokDir=/data1/home/igni1.5/
> work/ipc/shmem/e052dc07-8992-4834-87fd-abf84c0f644a-14472]
> [17:04:00,628][INFO][main][TcpCommunicationSpi] Successfully bound
> shared memory communication to TCP port [port=48104, locHost=
> 0.0.0.0/0.0.0.0]
> [17:04:00,652][INFO][main][TcpCommunicationSpi] Successfully bound to
> TCP port [port=47106, locHost=0.0.0.0/0.0.0.0]
> [17:04:00,685][WARNING][main][NoopCheckpointSpi] Checkpoints are
> disabled (to enable configure any GridCheckpointSpi implementation)
> [17:04:00,749][WARNING][main][GridCollisionManager] Collision
> resolution is disabled (all jobs will be activated upon arrival).
> [17:04:00,755][WARNING][main][NoopSwapSpaceSpi] Swap space is
> disabled. To enable use FileSwapSpaceSpi.
> [17:04:00,757][INFO][main][IgniteKernal%grid_recmd] Security status
> [authentication=off, tls/ssl=off]
> [17:04:00,861][WARNING][main][GridCacheProcessor] Deployment mode for
> cache is not CONTINUOUS or SHARED (it is recommended that you change
> deployment mode and restart): ISOLATED
> [17:04:01,040][INFO][main][GridTcpRestProtocol] Command protocol
> successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11217]
> [17:04:01,138][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi]
> Exception on joining: Connection refused
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1266)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1241)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> sendJoinRequest(ClientImpl.java:579)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> joinTopology(ClientImpl.java:505)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> access$900(ClientImpl.java:118)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.tryJoin(ClientImpl.java:1508)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1327)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:04:01,139][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi]
> Exception on joining: Connection refused
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1266)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1241)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> sendJoinRequest(ClientImpl.java:579)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> joinTopology(ClientImpl.java:505)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> access$900(ClientImpl.java:118)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.tryJoin(ClientImpl.java:1508)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1327)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:04:01,140][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi]
> Exception on joining: Connection refused
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1266)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1241)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> sendJoinRequest(ClientImpl.java:579)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> joinTopology(ClientImpl.java:505)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> access$900(ClientImpl.java:118)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.tryJoin(ClientImpl.java:1508)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1327)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:04:01,141][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi]
> Exception on joining: Connection refused
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1266)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.
> openSocket(TcpDiscoverySpi.java:1241)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> sendJoinRequest(ClientImpl.java:579)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> joinTopology(ClientImpl.java:505)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.
> access$900(ClientImpl.java:118)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.tryJoin(ClientImpl.java:1508)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1327)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:05:01,179][SEVERE][main][IgniteKernal%grid_recmd] Failed to start
> manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.
> discovery.GridDiscoveryManager]
> class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
> TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
> reconCnt=10, maxAckTimeout=600000, forceSrvMode=false,
> clientReconnectDisabled=false]
> at org.apache.ignite.internal.managers.GridManagerAdapter.
> startSpi(GridManagerAdapter.java:258)
> at org.apache.ignite.internal.managers.discovery.
> GridDiscoveryManager.start(GridDiscoveryManager.java:660)
> at org.apache.ignite.internal.IgniteKernal.startManager(
> IgniteKernal.java:1505)
> at org.apache.ignite.internal.IgniteKernal.start(
> IgniteKernal.java:917)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(
> IgnitionEx.java:1688)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(
> IgnitionEx.java:1547)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
> at org.apache.ignite.internal.IgnitionEx.startConfigurations(
> IgnitionEx.java:929)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
> at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
> Caused by: class org.apache.ignite.spi.IgniteSpiException: Join
> process timed out, did not receive response for join request (consider
> increasing 'joinTimeout' configuration property) [joinTimeout=60000,
> sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1334)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:05:01,180][SEVERE][main][IgniteKernal%grid_recmd] Got exception
> while starting (will rollback startup routine).
> class org.apache.ignite.IgniteCheckedException: Failed to start
> manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.
> internal.managers.discovery.GridDiscoveryManager]
> at org.apache.ignite.internal.IgniteKernal.startManager(
> IgniteKernal.java:1510)
> at org.apache.ignite.internal.IgniteKernal.start(
> IgniteKernal.java:917)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(
> IgnitionEx.java:1688)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(
> IgnitionEx.java:1547)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
> at org.apache.ignite.internal.IgnitionEx.startConfigurations(
> IgnitionEx.java:929)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
> at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000,
> ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false,
> clientReconnectDisabled=false]
> at org.apache.ignite.internal.managers.GridManagerAdapter.
> startSpi(GridManagerAdapter.java:258)
> at org.apache.ignite.internal.managers.discovery.
> GridDiscoveryManager.start(GridDiscoveryManager.java:660)
> at org.apache.ignite.internal.IgniteKernal.startManager(
> IgniteKernal.java:1505)
> ... 12 more
> Caused by: class org.apache.ignite.spi.IgniteSpiException: Join
> process timed out, did not receive response for join request (consider
> increasing 'joinTimeout' configuration property) [joinTimeout=60000,
> sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1334)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:05:01,193][INFO][main][GridTcpRestProtocol] Command protocol
> successfully stopped: TCP binary
> [17:05:01,201][WARNING][grid-nio-worker-0-#58%grid_recmd%][TcpCommunicationSpi]
> Local node is not started or fully initialized [isStopping=true]
> [17:05:01,201][WARNING][grid-nio-worker-1-#59%grid_recmd%][TcpCommunicationSpi]
> Local node is not started or fully initialized [isStopping=true]
> [17:05:01,843][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi]
> Failed to read message [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836],
> locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a,
> rmtNodeId=198bc1a5-1b1d-4b65-bccb-b2eb5ad1ec9b]
> class org.apache.ignite.IgniteCheckedException: Failed to deserialize
> object with given class loader: sun.misc.Launcher$AppClassLoader@3415ddf5
> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(
> JdkMarshaller.java:105)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> SocketReader.body(ClientImpl.java:888)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Caused by: java.io.IOException: Stream closed.
> at java.net.AbstractPlainSocketImpl.available(
> AbstractPlainSocketImpl.java:458)
> at java.net.SocketInputStream.available(SocketInputStream.java:245)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
> at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrappe
> r.read(JdkMarshallerInputStreamWrapper.java:53)
> at java.io.ObjectInputStream$PeekInputStream.read(
> ObjectInputStream.java:2310)
> at java.io.ObjectInputStream$PeekInputStream.readFully(
> ObjectInputStream.java:2323)
> at java.io.ObjectInputStream$BlockDataInputStream.
> readUTFBody(ObjectInputStream.java:3063)
> at java.io.ObjectInputStream$BlockDataInputStream.readUTF(
> ObjectInputStream.java:2864)
> at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.ignite.internal.util.IgniteUtils.readMap(
> IgniteUtils.java:4890)
> at org.apache.ignite.spi.discovery.tcp.internal.
> TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
> at java.io.ObjectInputStream.readExternalData(
> ObjectInputStream.java:1837)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1796)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at java.util.ArrayList.readObject(ArrayList.java:771)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(
> ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(
> ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
> at java.util.TreeMap.readObject(TreeMap.java:3234)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(
> ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(
> ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(
> JdkMarshaller.java:102)
> ... 2 more
> [17:05:01,845][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi]
> Connection failed [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836],
> locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a]
> java.io.IOException: Stream closed.
> at java.net.AbstractPlainSocketImpl.available(
> AbstractPlainSocketImpl.java:458)
> at java.net.SocketInputStream.available(SocketInputStream.java:245)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
> at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrappe
> r.read(JdkMarshallerInputStreamWrapper.java:53)
> at java.io.ObjectInputStream$PeekInputStream.read(
> ObjectInputStream.java:2310)
> at java.io.ObjectInputStream$PeekInputStream.readFully(
> ObjectInputStream.java:2323)
> at java.io.ObjectInputStream$BlockDataInputStream.
> readUTFBody(ObjectInputStream.java:3063)
> at java.io.ObjectInputStream$BlockDataInputStream.readUTF(
> ObjectInputStream.java:2864)
> at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.ignite.internal.util.IgniteUtils.readMap(
> IgniteUtils.java:4890)
> at org.apache.ignite.spi.discovery.tcp.internal.
> TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
> at java.io.ObjectInputStream.readExternalData(
> ObjectInputStream.java:1837)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1796)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at java.util.ArrayList.readObject(ArrayList.java:771)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(
> ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(
> ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
> at java.util.TreeMap.readObject(TreeMap.java:3234)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(
> ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(
> ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(
> ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(
> ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(
> JdkMarshaller.java:102)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> SocketReader.body(ClientImpl.java:888)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> [17:05:01,876][INFO][main][IgniteKernal%grid_recmd]
>
> >>> +-----------------------------------------------------------
> ----------------------------+
> >>> Ignite ver. 1.5.0-final#20151229-sha1:
> f1f8cda2f3f62231f42a59951bf34c39577c1bec stopped OK
> >>> +-----------------------------------------------------------
> ----------------------------+
> >>> Grid name: grid_recmd
> >>> Grid uptime: 00:01:02:580
>
>
> class org.apache.ignite.IgniteException: Failed to start manager:
> GridManagerAdapter [enabled=true, name=org.apache.ignite.
> internal.managers.discovery.GridDiscoveryManager]
> at org.apache.ignite.internal.util.IgniteUtils.
> convertException(IgniteUtils.java:882)
> at org.apache.ignite.Ignition.start(Ignition.java:350)
> at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
> at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.
> internal.managers.discovery.GridDiscoveryManager]
> at org.apache.ignite.internal.IgniteKernal.startManager(
> IgniteKernal.java:1510)
> at org.apache.ignite.internal.IgniteKernal.start(
> IgniteKernal.java:917)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(
> IgnitionEx.java:1688)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(
> IgnitionEx.java:1547)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
> at org.apache.ignite.internal.IgnitionEx.startConfigurations(
> IgnitionEx.java:929)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> ... 2 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000,
> ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false,
> clientReconnectDisabled=false]
> at org.apache.ignite.internal.managers.GridManagerAdapter.
> startSpi(GridManagerAdapter.java:258)
> at org.apache.ignite.internal.managers.discovery.
> GridDiscoveryManager.start(GridDiscoveryManager.java:660)
> at org.apache.ignite.internal.IgniteKernal.startManager(
> IgniteKernal.java:1505)
> ... 12 more
> Caused by: class org.apache.ignite.spi.IgniteSpiException: Join
> process timed out, did not receive response for join request (consider
> increasing 'joinTimeout' configuration property) [joinTimeout=60000,
> sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$
> MessageWorker.body(ClientImpl.java:1334)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
> the ignite process's JVM_OPTS is :
>
> export JVM_OPTS="-Xms5g -Xmx5g -Xmn2g -server -XX:-UseGCOverheadLimit
> -XX:+AggressiveOpts -XX:MaxPermSize=256m -XX:+UseP
> arNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB"
>
> the srv.xml is
>
> <bean id="cacheA" class="org.apache.ignite.configuration.
> CacheConfiguration">
> <property name="name" value="cache_recmd"/>
> <property name="cacheMode" value="PARTITIONED"/><!-- partition
> -->
> <property name="affinity">
> <bean class="org.apache.ignite.cache.affinity.rendezvous.
> RendezvousAffinityFunction">
> <property name="hashIdResolver" >
> <bean class="org.apache.ignite.cache.affinity.
> AffinityNodeAddressHashResolver"/>
> </property>
> <property name="partitions" value="280" />
> </bean>
> </property>
> <property name="backups" value="0"/>
> <property name="writeSynchronizationMode"
> value="PRIMARY_SYNC"/>
> <property name="atomicityMode" value="ATOMIC"/>
> <property name="atomicWriteOrderMode" value="PRIMARY"/>
> <property name="memoryMode" value="ONHEAP_TIERED"/><!-- memory
> tire -->
> <property name="startSize" value="#{30 * 1024 * 1024}"/>
> <property name="offHeapMaxMemory" value="#{512 * 1024 *
> 1024}"/>
> <property name="swapEnabled" value="true"/>
> <property name="evictionPolicy">
> <bean class="org.apache.ignite.cache.eviction.lru.
> LruEvictionPolicy">
> <property name="maxMemorySize" value="#{1024 * 1024 *
> 1024}"/>
> </bean>
> </property>
> <property name="rebalanceMode" value="ASYNC"/>
> <property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
> <property name="rebalanceThrottle" value="100"/>
> </bean>
>
> <bean id="igniA" class="org.apache.ignite.configuration.IgniteConfiguration"
> scope="singleton">
> <property name="gridName" value="grid_recmd"/>
> <property name="clientMode" value="false"/>
> <property name="metricsLogFrequency" value="0"/><!-- 0 means
> disable metrics log -->
> <property name="deploymentMode" value="ISOLATED"/>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.
> discovery.tcp.TcpDiscoverySpi">
> <property name="localPort" value="47500"/>
> <property name="localPortRange" value="20"/>
> <property name="ipFinder">
> <bean class="org.apache.ignite.spi.
> discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="shared" value="true"/>
> <property name="addresses">
> <list>
> <value>10.0.0.1:47500..47520</value>
> <value>10.0.0.2:47500..47520</value>
> </list>
> </property>
> </bean>
> </property>
> <property name="joinTimeout" value="60000"/>
> </bean>
> </property>
> <property name="communicationSpi">
> <bean class="org.apache.ignite.spi.communication.tcp.
> TcpCommunicationSpi">
> <property name="socketWriteTimeout" value="30000"/>
> <property name="connectTimeout" value="50000"/>
> </bean>
> </property>
> <property name="failureDetectionTimeout" value="60000"/>
> <property name="swapSpaceSpi">
> <bean class="org.apache.ignite.spi.swapspace.file.
> FileSwapSpaceSpi">
> <property name="baseDirectory"
> value="/data1/home/igni1.5/my/swp"/>
> </bean>
> </property>
> <property name="cacheConfiguration">
> <list>
> <ref bean="cacheA"/>
> </list>
> </property>
> <property name="includeEventTypes">
> <list>
> </list>
> </property>
> </bean>
>
>
--
Vladislav Pyatkov