You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ilya Suntsov (JIRA)" <ji...@apache.org> on 2015/09/15 12:08:45 UTC
[jira] [Assigned] (IGNITE-1490) Memory leaks during load testing (+
nearcache)
[ https://issues.apache.org/jira/browse/IGNITE-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ilya Suntsov reassigned IGNITE-1490:
------------------------------------
Assignee: Denis Magda (was: Ilya Suntsov)
> Memory leaks during load testing (+ nearcache)
> ----------------------------------------------
>
> Key: IGNITE-1490
> URL: https://issues.apache.org/jira/browse/IGNITE-1490
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: ignite-1.4
> Environment: Cent OS 2.6.32-573.3.1.el6.x86_64
> Reporter: Ilya Suntsov
> Assignee: Denis Magda
> Priority: Blocker
> Fix For: ignite-1.4
>
> Attachments: logs.zip, yardstick_config
>
>
> I have run atomic-put-get benchmark with nearcache (yardstick's config in attachment) on 5 hosts (13 nodes): 1 host - driver and on each of 4 servers 3 nodes.
> ~ 10 min after the start I got exception on driver node:
> {noformat}
> WARNING: Failed to process selector key (will close): GridSelectorNioSessionImpl [selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0 lim=4487 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], recovery=GridNioRecoveryDescriptor [acked=5431488, resendCnt=0, rcvCnt=5644052, reserved=true,
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Closing NIO session because of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:47101, failureDetectionTimeout=10000]
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[fosters-223/10.20.0.223:47101, /127.0.0.1:47101]]
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node is out of topology (probably, due to short-time network problems).
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Local node SEGMENTED: TcpDiscoveryNode [id=d4f641d6-2c0e-4547-b357-79fefe15cd4f, addrs=[10.20.0.218, 127.0.0.1], sockAddrs=[fosters-218/10.20.0.218:47500, /10.20.0.218:47500, /127.0.0.1:47500], discPort=47500, order=13, intOrder=13, lastExchangeTime=1442243902948, loc=true, ver=1.4.0#20150914-sha1:e5f16818, isClient=fa
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node is out of topology (probably, due to short-time network problems).
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Stopping local node according to configured segmentation policy.
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=135b3db6-ac16-4fdf-aa79-846ecc194b0a, addrs=[10.20.0.219, 127.0.0.1], sockAddrs=[fosters-219/10.20.0.219:47500, /10.20.0.219:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1442243211629, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger info
> INFO: Topology snapshot [ver=25, servers=1, clients=0, CPUs=16, heap=10.0GB]
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger info
> INFO: Command protocol successfully stopped: TCP binary
> Finishing main test [ts=1442243902993, date=Mon Sep 14 08:18:22 PDT 2015]
> ERROR: Shutting down benchmark driver to unexpected exception.
> Type '--help' for usage.
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to wait for topology update, cache (or node) is stopping.
> <------>at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1621)
> <------>at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1744)
> <------>at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:810)
> <------>at org.apache.ignite.yardstick.cache.IgnitePutGetBenchmark.test(IgnitePutGetBenchmark.java:32)
> <------>at org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:165)
> <------>at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to wait for topology update, cache (or node) is stopping.
> <------>at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.onKernalStop0(GridCacheAffinityManager.java:63)
> <------>at org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.onKernalStop(GridCacheManagerAdapter.java:110)
> <------>at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:1223)
> <------>at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:945)
> <------>at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1822)
> <------>at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1766)
> <------>at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
> <------>at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
> <------>at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:314)
> <------>at org.apache.ignite.Ignition.stop(Ignition.java:223)
> <------>at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$9.run(GridDiscoveryManager.java:1771)
> <------>... 1 more
> Sep 14, 2015 8:18:22 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=980f4cb7-e117-4877-89a0-9a1fee71a203, addrs=[10.20.0.219, 127.0.0.1], sockAddrs=[fosters-219/10.20.0.219:47501, /10.20.0.219:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1442243211629, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=1bf8ee2a-3bec-422b-8711-6a6b44933ae7, addrs=[10.20.0.219, 127.0.0.1], sockAddrs=[fosters-219/10.20.0.219:47502, /10.20.0.219:47502, /127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1442243211629, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=569473dc-4235-4008-9079-8fb9bf7c341c, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47500, /10.20.0.221:47500, /127.0.0.1:47500], discPort=47500, order=4, intOrder=4, lastExchangeTime=1442243211639, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=daea385c-156d-4264-ba55-7c506ab02f61, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47501, /10.20.0.221:47501, /127.0.0.1:47501], discPort=47501, order=5, intOrder=5, lastExchangeTime=1442243211639, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=c514ed85-8884-495a-9e0e-4ddfa49c851e, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47502, /10.20.0.221:47502, /127.0.0.1:47502], discPort=47502, order=6, intOrder=6, lastExchangeTime=1442243211639, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=c411fe74-3a76-4482-9d99-71cf8c3177f1, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47500, /10.20.0.222:47500, /127.0.0.1:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1442243211639, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=0872d78a-1843-421c-a65f-e1c4dee5e1da, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47501, /10.20.0.222:47501, /127.0.0.1:47501], discPort=47501, order=8, intOrder=8, lastExchangeTime=1442243211639, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=28603283-def1-495f-a665-b5501c13d272, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47502, /10.20.0.222:47502, /127.0.0.1:47502], discPort=47502, order=9, intOrder=9, lastExchangeTime=1442243211649, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=4e62c7a0-f5cb-4116-b9b4-325c3c48e39e, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47500, /10.20.0.223:47500, /127.0.0.1:47500], discPort=47500, order=10, intOrder=10, lastExchangeTime=1442243211649, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=2525979b-8801-4972-a6b8-c5aec51e5081, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47501, /10.20.0.223:47501, /127.0.0.1:47501], discPort=47501, order=11, intOrder=11, lastExchangeTime=1442243211649, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger warning
> WARNING: Node FAILED: TcpDiscoveryNode [id=199ceddf-ef56-4eb3-a1ab-d40385ad4887, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47502, /10.20.0.223:47502, /127.0.0.1:47502], discPort=47502, order=12, intOrder=12, lastExchangeTime=1442243211649, loc=false, ver=1.4.0#20150914-sha1:e5f16818, isClient=false]
> Sep 14, 2015 8:18:23 AM org.apache.ignite.logger.java.JavaLogger info
> {noformat}
> In GC log may notice a memory leak.
> Logs driver, servers and GC (from each host) in attachment
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)