You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Daniel López <d....@gmail.com> on 2016/06/02 08:21:59 UTC
One failing node stalling the whole cluster
Hi there,
We are using Ignite 1.5.0 and we are experiencing a strange issue where one
node stalls the others node in the cluster. We are
using CacheMode.REPLICATED caches to store data on heap on several nodes to
improve latency.
In one of the latest upgrades someone introduced a bug in the system that
could cause one node to consume too much memory and start having GC issues.
Sh*t happens :).
The problem, however, is that when this node starts to crawl due to heavy
GC usage, it starts spitting these logs:
|Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=24, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=284144, resendCnt=0,
rcvCnt=284230, reserved=true, lastAck=284224, nodeLeft=false,
node=TcpDiscoveryNode [id=1109a421-ec72-4534-99c4-df5d7e4f6136,
addrs=[x.y.z4, 127.0.0.1], sockAddrs=[machine.env/x.y.z4:3808, /x.y.z:3808,
/127.0.0.1:3808], discPort=3808, order=32, intOrder=17,
lastExchangeTime=1464854224616, loc=false,
ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], connected=true,
connectCnt=0, queueLimit=5120], super=GridNioSessionImpl
[locAddr=/x.y.z3:47100, rmtAddr=/x.y.z4:33450, createTime=1464854224707,
closeTime=0, bytesSent=10838655, bytesRcvd=221207982,
sndSchedTime=1464854575433, lastSndTime=1464854575433,
lastRcvTime=1464854575433, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@63f44e30, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]
WARN |2016-06-02T08:03:02,575||TcpCommunicationSpi|Closing NIO session
because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Conexión reinicializada por la
máquina remota]
And the other nodes in the cluster start to produce these other logs and
access to the cache slows down/pauses greatly:
GridCachePartitionExchangeManager|Failed to send partitions full message
[node=TcpDiscoveryNode [id=913ea465-ed45-4ec9-a4b7-d2c5f9c57a2e, a....
TcpDiscoverySpi|Failed to ping node (status check will be initiated): ....
GridDiscoveryManager|Node FAILED: TcpDiscoveryNode [id=...
That the node with the GC issues stops working is "normal", even if
undesired, but what really worries us is that it causes the other nodes in
the cluster to stop being able to use the replicated caches, so one node
can bring down the whole cluster.
If we stop the offeding node, the others go back to normal behaviour and
work as fast as always.
We are going to solve the application bug, of course, but is there any
configuration setting that we can tweak so one bug in one machine does not
bring the whole cluster to a halt?
Ignite is configured to use TcpDiscoverySpi with TcpDiscoveryVmIpFinder
with a list of addresses (11 nodes per set currently)
Each node has 29 caches configured like this:
cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
cacheConfiguration.setCopyOnRead(false);
cacheConfiguration.setEagerTtl(false);
Thanks,
D.
PD: Yes, we'll have to try with latest Ignite version but we wanted to know
if there is any configuration setting that might help first, before having
to migrate and restart the whole testing/process.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
Hi Binti,
Is the second application that experiences GC pauses due to batch related activity embeds an Ignite server node inside of its process?
If it’s so then when the first application tries to update a replicated cache waits while it’s updated on the node that is a part of the second application process.
But if it’s not the case then please provide full details of your cluster and make thread dumps from all the nodes during the freezing.
—
Denis
> On Aug 19, 2016, at 4:31 PM, bintisepaha <bi...@tudor.com> wrote:
>
> Hi, we are seeing similar issues too with ignite 1.5.0. Were you guys able to
> resolve it? we use distributed caches in full-sync mode and also a few
> replicated caches.
>
> Thanks,
> Binti
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7183.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@apache.org>.
Vladimir Ozerov as far as I recall you have faced with the issue when a slow client affected performance of the whole cluster recently. Please chime in this discussion confirming that the symptoms are the same. Most likely you already created a JIRA ticket for this issue.
> Also it's not clear why the nodes would require to GC because all the caches
> are held off-heap and we have a 10G heap running G1GC.
Your logic can fill up Java heap with temporal objects. Every time you need to get a value from cache its copy is moved from off_heap to Java heap for a period of time until your code holds a reference to it. This is related to all kind of queries. You need to get heap dumps and/or heap histogram to see what kind of objects are in your heap.
—
Denis
> On Sep 16, 2016, at 6:47 AM, yfernando <yo...@tudor.com> wrote:
>
> Thanks for your reply Anmol. Do you know if there is a bug logged against
> this which we can track?
>
> Also it's not clear why the nodes would require to GC because all the caches
> are held off-heap and we have a 10G heap running G1GC.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7799.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by DLopez <d....@gmail.com>.
FWIW, in our case, the GC was not the problem with Ignite. The heap issue
was already diagnosed, well known and unrelated. The problem was that the
slow down in one node was causing all the other nodes in the grid to
basically lock when reading from the cache, without suffering any GC issue.
A simple problem in one node should not be able to cause your grid to go
down, specially when talking about replicated caches.
GC issues, unrelated to Ignite, will probably appear associated with the
problem because it's one of the most common ways a Java app slows down
without dying completely.
My 2ec
D.
2016-09-16 18:19 GMT+02:00 Ignitebie [via Apache Ignite Users] <
ml-node+s70518n7807h37@n6.nabble.com>:
> That would be topic for discussion on how off heap actually work. My
> understanding is to start with object creation will happen on heap (YG)
> and then moved to Old or Off heap.
>
> If allocation of object creation (I believe after that only you will
> associating them in a map (key.value cache), is quit high, say larger than
> YG size , per sec, then application will be engaged in GC triggered due to
> not being able to allocation.
>
>
> Do you see GC trigerred for YG and messages such as Allocation Failure.
> Have you enabled GC logging.
>
>
>
> On Fri, Sep 16, 2016 at 2:47 PM, yfernando <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7807&i=0>> wrote:
>
>> Thanks for your reply Anmol. Do you know if there is a bug logged against
>> this which we can track?
>>
>> Also it's not clear why the nodes would require to GC because all the
>> caches
>> are held off-heap and we have a 10G heap running G1GC.
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/One-failing-node-stalling-the-whole-
>> cluster-tp5372p7799.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/One-
> failing-node-stalling-the-whole-cluster-tp5372p7807.html
> To unsubscribe from One failing node stalling the whole cluster, click
> here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5372&code=ZC5sb3Blei5qQGdtYWlsLmNvbXw1MzcyfDIwNTkzNjQ3OTE=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7808.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by vkulichenko <va...@gmail.com>.
Hi Sparkle,
Please properly subscribe to the mailing list so that the community can
receive email notifications for your messages. To subscribe, send empty
email to user-subscribe@ignite.apache.org and follow simple instructions in
the reply.
sparkle_j wrote
> In one of our tests, we noticed that Ignite's TcpCommunicationSpi object
> is growing and retaining over 70% of heap memory. Not sure why. Is this a
> known issue..? Please let us know if this is already addressed.
>
> We are testing with Ignite1.5, 10GB heap, with moderate amount of data on
> start up and just connect and disconnect dummy clients to grid. This is
> causing the heap to grow over time and eventually Full GC and node is
> being killed by Ignite. This is also causing cluster wide instability.
> Please see HeapDump screenshots attached.
>
> TcpCommSpi.png
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7879/TcpCommSpi.png>
>
> dom-tree-1.png
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7879/dom-tree-1.png>
>
> dom-tree.png
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7879/dom-tree.png>
Can you take the full heap dump and upload it somewhere?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7887.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Anmol Rattan <an...@gmail.com>.
That would be topic for discussion on how off heap actually work. My
understanding is to start with object creation will happen on heap (YG)
and then moved to Old or Off heap.
If allocation of object creation (I believe after that only you will
associating them in a map (key.value cache), is quit high, say larger than
YG size , per sec, then application will be engaged in GC triggered due to
not being able to allocation.
Do you see GC trigerred for YG and messages such as Allocation Failure.
Have you enabled GC logging.
On Fri, Sep 16, 2016 at 2:47 PM, yfernando <yo...@tudor.com> wrote:
> Thanks for your reply Anmol. Do you know if there is a bug logged against
> this which we can track?
>
> Also it's not clear why the nodes would require to GC because all the
> caches
> are held off-heap and we have a 10G heap running G1GC.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/One-failing-node-stalling-the-
> whole-cluster-tp5372p7799.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: One failing node stalling the whole cluster
Posted by yfernando <yo...@tudor.com>.
Thanks for your reply Anmol. Do you know if there is a bug logged against
this which we can track?
Also it's not clear why the nodes would require to GC because all the caches
are held off-heap and we have a 10G heap running G1GC.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7799.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Anmol Rattan <an...@gmail.com>.
That is known error at least in 1.6. I am not sure a fix for this is even
in 1.7. For gc pause, if there are actually any, worth considering jvm
tuning and seeing allocation and promotion rate.
In our case, we had to increase younger gen to have 8GB space to deal
with.
However, slow client definitely hang whole grid, even if there are no GC,
A chicken egg problem results. If you increase timeout, grid hangs for
longer time.
if your reduce timeout, clients/nodes will leave grid early and even go in
segmentation and Segmentation policy handling via starting ignite bean only
works if you start process with ignite script. If prcoess has been started
otherwise in a custom script, it does not support.
Thanks & Regards
Anmol Rattan
+91 9538901262
On Fri, Sep 16, 2016 at 10:44 AM, yfernando <yo...@tudor.com>
wrote:
> Hi Denis,
>
> We have been able to reproduce this situation where a node failure freezes
> the entire grid.
>
> Please find the full thread dumps of the 5 nodes that are locked up.
>
> The memoryMode of the caches are configured to be OFFHEAP_TIERED
> The cacheMode is PARTITIONED
> The atomicityMode is TRANSACTIONAL
>
> We have also seen ALL the clients freeze during a FULL GC occurring on ANY
> single node.
>
> Please let us know if you require any more information.
>
> grid-tp1-dev-11220-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7791/grid-tp1-dev-11220-201609141523318.txt>
> grid-tp1-dev-11223-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7791/grid-tp1-dev-11223-201609141523318.txt>
> grid-tp3-dev-11220-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7791/grid-tp3-dev-11220-201609141523318.txt>
> grid-tp3-dev-11221-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7791/grid-tp3-dev-11221-201609141523318.txt>
> grid-tp4-dev-11220-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7791/grid-tp4-dev-11220-201609141523318.txt>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/One-failing-node-stalling-the-
> whole-cluster-tp5372p7791.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: One failing node stalling the whole cluster
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Ignite does not have 1.6.x versions.
It seems, you would better to ask GridGain support.
On Wed, Feb 8, 2017 at 6:05 AM, ght230 <gh...@163.com> wrote:
> I have met the same problem.
>
> It seems that IGNITE-4003 will be fixed in version 2.0.
>
> I am using version 1.6.8 now.
>
> I want to know whether the patch about IGNITE-4003 will be merged in
> Vesion
> 1.6.x?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/One-failing-node-stalling-the-
> whole-cluster-tp5372p10496.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@apache.org>.
Hi,
Presently Apache Ignite community releases only 1.x versions. The version you’re referring to should be used by other product built on top of Ignite.
—
Denis
> On Feb 7, 2017, at 7:05 PM, ght230 <gh...@163.com> wrote:
>
> I have met the same problem.
>
> It seems that IGNITE-4003 will be fixed in version 2.0.
>
> I am using version 1.6.8 now.
>
> I want to know whether the patch about IGNITE-4003 will be merged in Vesion
> 1.6.x?
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p10496.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by ght230 <gh...@163.com>.
I have met the same problem.
It seems that IGNITE-4003 will be fixed in version 2.0.
I am using version 1.6.8 now.
I want to know whether the patch about IGNITE-4003 will be merged in Vesion
1.6.x?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p10496.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
A correct link is the following
https://issues.apache.org/jira/browse/IGNITE-4003 <https://issues.apache.org/jira/browse/IGNITE-4003>
> On Sep 29, 2016, at 9:31 AM, Denis Magda <dm...@gridgain.com> wrote:
>
> Good news to everyone. Looks like we could get to the bottom of this issue
> https://ggsystems.atlassian.net/browse/IGN-5958 <https://ggsystems.atlassian.net/browse/IGN-5958>
>
> Hope it will be fixed soon.
>
> —
> Denis
>
>> On Sep 16, 2016, at 9:38 AM, yfernando <yohan.fernando@tudor.com <ma...@tudor.com>> wrote:
>>
>> Unfortunately iam unable to send the full logs files but they contain the
>> following exceptions
>>
>> [14 Sep 2016 11:14:30.290 EDT] [pub-#16%DataGridServer-Development%] ERROR
>> 11223 (OrderHolderSaveRunnable.java:273) exception ocurred while generating
>> Trade Order for Order: OrderKey [traderId=5
>> 207, orderId=16084348]
>> javax.cache.CacheException: class
>> org.apache.ignite.transactions.TransactionTimeoutException: Failed to
>> acquire lock within provided timeout for transaction [timeout=5000,
>> tx=GridNearTxLocal [ma
>> ppings=IgniteTxMappingsImpl [], nearLocallyMapped=false,
>> colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false,
>> mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter [
>> nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
>> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
>> depEnabled=false, txState=IgniteTxStateImpl
>> [activeCacheIds=GridLongList [idx=1, arr=[1633849959]], txMap={IgniteTxKey
>> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
>> hasValBytes=true], cacheId=1633849959]=IgniteTxEntry [key=Ke
>> yCacheObjectImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
>> cacheId=1633849959, txKey=IgniteTxKey [key=KeyCacheObjectImpl
>> [val=BatchIdKey [privDb=trim_sys], hasValBytes=true], cacheId
>> =1633849959], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
>> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
>> explicitVer=null, dhtVer=null, filters=null, filters
>> Passed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
>> [super=GridDistributedCacheEntry [super=GridCacheMapEntry
>> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=tr
>> ue], val=null, startVer=1473869129773, ver=GridCacheVersion
>> [topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
>> order=1473869129773], hash=1508409679, extras=null, flags=0]]], prepared
>> =false, locked=false, nodeId=3cd37805-46a7-4287-875e-9cbd0cf03fad,
>> locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0,
>> partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [
>> topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
>> order=1473869129772]]}], super=IgniteTxAdapter [xidVer=GridCacheVersion
>> [topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
>> order=1473869129772], writeVer=null, implicit=false, loc=true, threadId=50,
>> startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
>> startVer=GridCacheVersion [topVer=85333522, nod
>> eOrderDrId=10, globalTime=1473859812640, order=1473869129772], endVer=null,
>> isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
>> sysInvalidate=false, sys=false, plc=2, commitVer=nul
>> l, finalizing=NONE, preparing=false, invalidParts=null,
>> state=MARKED_ROLLBACK, timedOut=false, topVer=AffinityTopologyVersion
>> [topVer=101, minorTopVer=0], duration=5007ms, onePhaseCommit=false],
>> size=1]]]]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1618)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1841)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:871)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> com.somecompany.grid.server.tradegen.BatchIdHelper.getListOfIds(BatchIdHelper.java:69)
>> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
>> at
>> com.somecompany.grid.server.tradegen.TradeGenerator.generateUniqueTradeId64(TradeGenerator.java:47)
>> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
>> at
>> com.somecompany.grid.server.tradegen.TradeGenerator.allocateTradesFromFills(TradeGenerator.java:158)
>> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
>> at
>> com.somecompany.grid.server.tradegen.OrderHolderSaveRunnable.run(OrderHolderSaveRunnable.java:271)
>> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
>> at
>> org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1879)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6397)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1166)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1770)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [?:1.8.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [?:1.8.0_60]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
>> Caused by: org.apache.ignite.transactions.TransactionTimeoutException:
>> Failed to acquire lock within provided timeout for transaction
>> [timeout=5000, tx=GridNearTxLocal [mappings=IgniteTxMappings
>> Impl [], nearLocallyMapped=false, colocatedLocallyMapped=false,
>> needCheckBackup=null, hasRemoteLocks=false, mappings=IgniteTxMappingsImpl
>> [], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=f
>> alse, nearNodes=[], dhtNodes=[], explicitLock=false,
>> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
>> depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=GridLon
>> gList [idx=1, arr=[1633849959]], txMap={IgniteTxKey [key=KeyCacheObjectImpl
>> [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
>> cacheId=1633849959]=IgniteTxEntry [key=KeyCacheObjectImpl [val=B
>> atchIdKey [privDb=trim_sys], hasValBytes=true], cacheId=1633849959,
>> txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
>> hasValBytes=true], cacheId=1633849959], val=[op=R
>> EAD, val=null], prevVal=[op=NOOP, val=null], entryProcessorsCol=null,
>> ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null,
>> dhtVer=null, filters=null, filtersPassed=false, filtersSe
>> t=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry
>> [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=BatchIdKey
>> [privDb=trim_sys], hasValBytes=true], val=null, startVer
>> =1473869129773, ver=GridCacheVersion [topVer=85333522, nodeOrderDrId=10,
>> globalTime=1473859812640, order=1473869129773], hash=1508409679,
>> extras=null, flags=0]]], prepared=false, locked=false, n
>> odeId=3cd37805-46a7-4287-875e-9cbd0cf03fad, locMapped=false, expiryPlc=null,
>> transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
>> xidVer=GridCacheVersion [topVer=85333522, nodeOr
>> derDrId=10, globalTime=1473859812640, order=1473869129772]]}],
>> super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=85333522,
>> nodeOrderDrId=10, globalTime=1473859812640, order=1473869129772],
>> writeVer=null, implicit=false, loc=true, threadId=50,
>> startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
>> startVer=GridCacheVersion [topVer=85333522, nodeOrderDrId=10, globalTi
>> me=1473859812640, order=1473869129772], endVer=null,
>> isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
>> sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, pre
>> paring=false, invalidParts=null, state=MARKED_ROLLBACK, timedOut=false,
>> topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], duration=5007ms,
>> onePhaseCommit=false], size=1]]]]
>> at
>> org.apache.ignite.internal.util.IgniteUtils$12.apply(IgniteUtils.java:791)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.IgniteUtils$12.apply(IgniteUtils.java:789)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> ... 21 more
>> Caused by:
>> org.apache.ignite.internal.transactions.IgniteTxTimeoutCheckedException:
>> Failed to acquire lock within provided timeout for transaction
>> [timeout=5000, tx=GridNearTxLocal [mappings=Ign
>> iteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMapped=false,
>> needCheckBackup=null, hasRemoteLocks=false, mappings=IgniteTxMappingsImpl
>> [], super=GridDhtTxLocalAdapter [nearOnOrig
>> inatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
>> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
>> depEnabled=false, txState=IgniteTxStateImpl [activeCac
>> heIds=GridLongList [idx=1, arr=[1633849959]], txMap={IgniteTxKey
>> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
>> hasValBytes=true], cacheId=1633849959]=IgniteTxEntry [key=KeyCacheObje
>> ctImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
>> cacheId=1633849959, txKey=IgniteTxKey [key=KeyCacheObjectImpl
>> [val=BatchIdKey [privDb=trim_sys], hasValBytes=true], cacheId=163384995
>> 9], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
>> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
>> explicitVer=null, dhtVer=null, filters=null, filtersPassed=fal
>> se, filtersSet=true, entry=GridDhtDetachedCacheEntry
>> [super=GridDistributedCacheEntry [super=GridCacheMapEntry
>> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
>> hasValBytes=true], val=n
>> ull, startVer=1473869129773, ver=GridCacheVersion [topVer=85333522,
>> nodeOrderDrId=10, globalTime=1473859812640, order=1473869129773],
>> hash=1508409679, extras=null, flags=0]]], prepared=false, lo
>> cked=false, nodeId=3cd37805-46a7-4287-875e-9cbd0cf03fad, locMapped=false,
>> expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
>> serReadVer=null, xidVer=GridCacheVersion [topVer=853
>> 33522, nodeOrderDrId=10, globalTime=1473859812640, order=1473869129772]]}],
>> super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=85333522,
>> nodeOrderDrId=10, globalTime=1473859812640, order=147
>> 3869129772], writeVer=null, implicit=false, loc=true, threadId=50,
>> startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
>> startVer=GridCacheVersion [topVer=85333522, nodeOrderDrId
>> =10, globalTime=1473859812640, order=1473869129772], endVer=null,
>> isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
>> sysInvalidate=false, sys=false, plc=2, commitVer=null, finaliz
>> ing=NONE, preparing=false, invalidParts=null, state=MARKED_ROLLBACK,
>> timedOut=false, topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
>> duration=5007ms, onePhaseCommit=false], size=1]]]
>> ]
>> at
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter$PostLockClosure2.apply(IgniteTxLocalAdapter.java:4023)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter$PostLockClosure2.apply(IgniteTxLocalAdapter.java:4010)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridEmbeddedFuture$3.applyx(GridEmbeddedFuture.java:158)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:297)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:290)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:262)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:250)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:380)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:346)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.onComplete(GridDhtColocatedLockFuture.java:535)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.fi <http://1.5.0.fi/>
>> nal]
>> at
>> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.access$1100(GridDhtColocatedLockFuture.java:78)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.fi <http://1.5.0.fi/>
>> nal]
>> at
>> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture$LockTimeoutObject.onTimeout(GridDhtColocatedLockFuture.java:1291)
>> ~[ignite-core-1.5.0.
>> final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:159)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>> at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7809.html <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7809.html>
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com <http://nabble.com/>.
>
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
Good news to everyone. Looks like we could get to the bottom of this issue
https://ggsystems.atlassian.net/browse/IGN-5958 <https://ggsystems.atlassian.net/browse/IGN-5958>
Hope it will be fixed soon.
—
Denis
> On Sep 16, 2016, at 9:38 AM, yfernando <yo...@tudor.com> wrote:
>
> Unfortunately iam unable to send the full logs files but they contain the
> following exceptions
>
> [14 Sep 2016 11:14:30.290 EDT] [pub-#16%DataGridServer-Development%] ERROR
> 11223 (OrderHolderSaveRunnable.java:273) exception ocurred while generating
> Trade Order for Order: OrderKey [traderId=5
> 207, orderId=16084348]
> javax.cache.CacheException: class
> org.apache.ignite.transactions.TransactionTimeoutException: Failed to
> acquire lock within provided timeout for transaction [timeout=5000,
> tx=GridNearTxLocal [ma
> ppings=IgniteTxMappingsImpl [], nearLocallyMapped=false,
> colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false,
> mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter [
> nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
> depEnabled=false, txState=IgniteTxStateImpl
> [activeCacheIds=GridLongList [idx=1, arr=[1633849959]], txMap={IgniteTxKey
> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
> hasValBytes=true], cacheId=1633849959]=IgniteTxEntry [key=Ke
> yCacheObjectImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
> cacheId=1633849959, txKey=IgniteTxKey [key=KeyCacheObjectImpl
> [val=BatchIdKey [privDb=trim_sys], hasValBytes=true], cacheId
> =1633849959], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
> explicitVer=null, dhtVer=null, filters=null, filters
> Passed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
> [super=GridDistributedCacheEntry [super=GridCacheMapEntry
> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=tr
> ue], val=null, startVer=1473869129773, ver=GridCacheVersion
> [topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
> order=1473869129773], hash=1508409679, extras=null, flags=0]]], prepared
> =false, locked=false, nodeId=3cd37805-46a7-4287-875e-9cbd0cf03fad,
> locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0,
> partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [
> topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
> order=1473869129772]]}], super=IgniteTxAdapter [xidVer=GridCacheVersion
> [topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
> order=1473869129772], writeVer=null, implicit=false, loc=true, threadId=50,
> startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
> startVer=GridCacheVersion [topVer=85333522, nod
> eOrderDrId=10, globalTime=1473859812640, order=1473869129772], endVer=null,
> isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
> sysInvalidate=false, sys=false, plc=2, commitVer=nul
> l, finalizing=NONE, preparing=false, invalidParts=null,
> state=MARKED_ROLLBACK, timedOut=false, topVer=AffinityTopologyVersion
> [topVer=101, minorTopVer=0], duration=5007ms, onePhaseCommit=false],
> size=1]]]]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1618)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1841)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:871)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> com.somecompany.grid.server.tradegen.BatchIdHelper.getListOfIds(BatchIdHelper.java:69)
> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
> at
> com.somecompany.grid.server.tradegen.TradeGenerator.generateUniqueTradeId64(TradeGenerator.java:47)
> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
> at
> com.somecompany.grid.server.tradegen.TradeGenerator.allocateTradesFromFills(TradeGenerator.java:158)
> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
> at
> com.somecompany.grid.server.tradegen.OrderHolderSaveRunnable.run(OrderHolderSaveRunnable.java:271)
> ~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1879)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6397)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1166)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1770)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
> Caused by: org.apache.ignite.transactions.TransactionTimeoutException:
> Failed to acquire lock within provided timeout for transaction
> [timeout=5000, tx=GridNearTxLocal [mappings=IgniteTxMappings
> Impl [], nearLocallyMapped=false, colocatedLocallyMapped=false,
> needCheckBackup=null, hasRemoteLocks=false, mappings=IgniteTxMappingsImpl
> [], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=f
> alse, nearNodes=[], dhtNodes=[], explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
> depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=GridLon
> gList [idx=1, arr=[1633849959]], txMap={IgniteTxKey [key=KeyCacheObjectImpl
> [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
> cacheId=1633849959]=IgniteTxEntry [key=KeyCacheObjectImpl [val=B
> atchIdKey [privDb=trim_sys], hasValBytes=true], cacheId=1633849959,
> txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
> hasValBytes=true], cacheId=1633849959], val=[op=R
> EAD, val=null], prevVal=[op=NOOP, val=null], entryProcessorsCol=null,
> ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null,
> dhtVer=null, filters=null, filtersPassed=false, filtersSe
> t=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry
> [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=BatchIdKey
> [privDb=trim_sys], hasValBytes=true], val=null, startVer
> =1473869129773, ver=GridCacheVersion [topVer=85333522, nodeOrderDrId=10,
> globalTime=1473859812640, order=1473869129773], hash=1508409679,
> extras=null, flags=0]]], prepared=false, locked=false, n
> odeId=3cd37805-46a7-4287-875e-9cbd0cf03fad, locMapped=false, expiryPlc=null,
> transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
> xidVer=GridCacheVersion [topVer=85333522, nodeOr
> derDrId=10, globalTime=1473859812640, order=1473869129772]]}],
> super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=85333522,
> nodeOrderDrId=10, globalTime=1473859812640, order=1473869129772],
> writeVer=null, implicit=false, loc=true, threadId=50,
> startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
> startVer=GridCacheVersion [topVer=85333522, nodeOrderDrId=10, globalTi
> me=1473859812640, order=1473869129772], endVer=null,
> isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
> sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, pre
> paring=false, invalidParts=null, state=MARKED_ROLLBACK, timedOut=false,
> topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], duration=5007ms,
> onePhaseCommit=false], size=1]]]]
> at
> org.apache.ignite.internal.util.IgniteUtils$12.apply(IgniteUtils.java:791)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.IgniteUtils$12.apply(IgniteUtils.java:789)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> ... 21 more
> Caused by:
> org.apache.ignite.internal.transactions.IgniteTxTimeoutCheckedException:
> Failed to acquire lock within provided timeout for transaction
> [timeout=5000, tx=GridNearTxLocal [mappings=Ign
> iteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMapped=false,
> needCheckBackup=null, hasRemoteLocks=false, mappings=IgniteTxMappingsImpl
> [], super=GridDhtTxLocalAdapter [nearOnOrig
> inatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
> super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
> depEnabled=false, txState=IgniteTxStateImpl [activeCac
> heIds=GridLongList [idx=1, arr=[1633849959]], txMap={IgniteTxKey
> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
> hasValBytes=true], cacheId=1633849959]=IgniteTxEntry [key=KeyCacheObje
> ctImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
> cacheId=1633849959, txKey=IgniteTxKey [key=KeyCacheObjectImpl
> [val=BatchIdKey [privDb=trim_sys], hasValBytes=true], cacheId=163384995
> 9], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
> explicitVer=null, dhtVer=null, filters=null, filtersPassed=fal
> se, filtersSet=true, entry=GridDhtDetachedCacheEntry
> [super=GridDistributedCacheEntry [super=GridCacheMapEntry
> [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
> hasValBytes=true], val=n
> ull, startVer=1473869129773, ver=GridCacheVersion [topVer=85333522,
> nodeOrderDrId=10, globalTime=1473859812640, order=1473869129773],
> hash=1508409679, extras=null, flags=0]]], prepared=false, lo
> cked=false, nodeId=3cd37805-46a7-4287-875e-9cbd0cf03fad, locMapped=false,
> expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
> serReadVer=null, xidVer=GridCacheVersion [topVer=853
> 33522, nodeOrderDrId=10, globalTime=1473859812640, order=1473869129772]]}],
> super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=85333522,
> nodeOrderDrId=10, globalTime=1473859812640, order=147
> 3869129772], writeVer=null, implicit=false, loc=true, threadId=50,
> startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
> startVer=GridCacheVersion [topVer=85333522, nodeOrderDrId
> =10, globalTime=1473859812640, order=1473869129772], endVer=null,
> isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
> sysInvalidate=false, sys=false, plc=2, commitVer=null, finaliz
> ing=NONE, preparing=false, invalidParts=null, state=MARKED_ROLLBACK,
> timedOut=false, topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
> duration=5007ms, onePhaseCommit=false], size=1]]]
> ]
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter$PostLockClosure2.apply(IgniteTxLocalAdapter.java:4023)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter$PostLockClosure2.apply(IgniteTxLocalAdapter.java:4010)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridEmbeddedFuture$3.applyx(GridEmbeddedFuture.java:158)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:297)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:290)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:262)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:250)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:380)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:346)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.onComplete(GridDhtColocatedLockFuture.java:535)
> ~[ignite-core-1.5.0.final.jar:1.5.0.fi
> nal]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.access$1100(GridDhtColocatedLockFuture.java:78)
> ~[ignite-core-1.5.0.final.jar:1.5.0.fi
> nal]
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture$LockTimeoutObject.onTimeout(GridDhtColocatedLockFuture.java:1291)
> ~[ignite-core-1.5.0.
> final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:159)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> ~[ignite-core-1.5.0.final.jar:1.5.0.final]
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7809.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by yfernando <yo...@tudor.com>.
Unfortunately iam unable to send the full logs files but they contain the
following exceptions
[14 Sep 2016 11:14:30.290 EDT] [pub-#16%DataGridServer-Development%] ERROR
11223 (OrderHolderSaveRunnable.java:273) exception ocurred while generating
Trade Order for Order: OrderKey [traderId=5
207, orderId=16084348]
javax.cache.CacheException: class
org.apache.ignite.transactions.TransactionTimeoutException: Failed to
acquire lock within provided timeout for transaction [timeout=5000,
tx=GridNearTxLocal [ma
ppings=IgniteTxMappingsImpl [], nearLocallyMapped=false,
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false,
mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter [
nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
depEnabled=false, txState=IgniteTxStateImpl
[activeCacheIds=GridLongList [idx=1, arr=[1633849959]], txMap={IgniteTxKey
[key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
hasValBytes=true], cacheId=1633849959]=IgniteTxEntry [key=Ke
yCacheObjectImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
cacheId=1633849959, txKey=IgniteTxKey [key=KeyCacheObjectImpl
[val=BatchIdKey [privDb=trim_sys], hasValBytes=true], cacheId
=1633849959], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=null, filters
Passed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=tr
ue], val=null, startVer=1473869129773, ver=GridCacheVersion
[topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
order=1473869129773], hash=1508409679, extras=null, flags=0]]], prepared
=false, locked=false, nodeId=3cd37805-46a7-4287-875e-9cbd0cf03fad,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [
topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
order=1473869129772]]}], super=IgniteTxAdapter [xidVer=GridCacheVersion
[topVer=85333522, nodeOrderDrId=10, globalTime=1473859812640,
order=1473869129772], writeVer=null, implicit=false, loc=true, threadId=50,
startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
startVer=GridCacheVersion [topVer=85333522, nod
eOrderDrId=10, globalTime=1473859812640, order=1473869129772], endVer=null,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
sysInvalidate=false, sys=false, plc=2, commitVer=nul
l, finalizing=NONE, preparing=false, invalidParts=null,
state=MARKED_ROLLBACK, timedOut=false, topVer=AffinityTopologyVersion
[topVer=101, minorTopVer=0], duration=5007ms, onePhaseCommit=false],
size=1]]]]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1618)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1841)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:871)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
com.somecompany.grid.server.tradegen.BatchIdHelper.getListOfIds(BatchIdHelper.java:69)
~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
at
com.somecompany.grid.server.tradegen.TradeGenerator.generateUniqueTradeId64(TradeGenerator.java:47)
~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
at
com.somecompany.grid.server.tradegen.TradeGenerator.allocateTradesFromFills(TradeGenerator.java:158)
~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
at
com.somecompany.grid.server.tradegen.OrderHolderSaveRunnable.run(OrderHolderSaveRunnable.java:271)
~[data-grid-server-ignite-3.0-SNAPSHOT.jar:3.0-SNAPSHOT]
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4.execute(GridClosureProcessor.java:1879)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6397)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1166)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1770)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: org.apache.ignite.transactions.TransactionTimeoutException:
Failed to acquire lock within provided timeout for transaction
[timeout=5000, tx=GridNearTxLocal [mappings=IgniteTxMappings
Impl [], nearLocallyMapped=false, colocatedLocallyMapped=false,
needCheckBackup=null, hasRemoteLocks=false, mappings=IgniteTxMappingsImpl
[], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=f
alse, nearNodes=[], dhtNodes=[], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=GridLon
gList [idx=1, arr=[1633849959]], txMap={IgniteTxKey [key=KeyCacheObjectImpl
[val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
cacheId=1633849959]=IgniteTxEntry [key=KeyCacheObjectImpl [val=B
atchIdKey [privDb=trim_sys], hasValBytes=true], cacheId=1633849959,
txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
hasValBytes=true], cacheId=1633849959], val=[op=R
EAD, val=null], prevVal=[op=NOOP, val=null], entryProcessorsCol=null,
ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null,
dhtVer=null, filters=null, filtersPassed=false, filtersSe
t=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry
[super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=BatchIdKey
[privDb=trim_sys], hasValBytes=true], val=null, startVer
=1473869129773, ver=GridCacheVersion [topVer=85333522, nodeOrderDrId=10,
globalTime=1473859812640, order=1473869129773], hash=1508409679,
extras=null, flags=0]]], prepared=false, locked=false, n
odeId=3cd37805-46a7-4287-875e-9cbd0cf03fad, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=85333522, nodeOr
derDrId=10, globalTime=1473859812640, order=1473869129772]]}],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=85333522,
nodeOrderDrId=10, globalTime=1473859812640, order=1473869129772],
writeVer=null, implicit=false, loc=true, threadId=50,
startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
startVer=GridCacheVersion [topVer=85333522, nodeOrderDrId=10, globalTi
me=1473859812640, order=1473869129772], endVer=null,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, pre
paring=false, invalidParts=null, state=MARKED_ROLLBACK, timedOut=false,
topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], duration=5007ms,
onePhaseCommit=false], size=1]]]]
at
org.apache.ignite.internal.util.IgniteUtils$12.apply(IgniteUtils.java:791)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.IgniteUtils$12.apply(IgniteUtils.java:789)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
... 21 more
Caused by:
org.apache.ignite.internal.transactions.IgniteTxTimeoutCheckedException:
Failed to acquire lock within provided timeout for transaction
[timeout=5000, tx=GridNearTxLocal [mappings=Ign
iteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMapped=false,
needCheckBackup=null, hasRemoteLocks=false, mappings=IgniteTxMappingsImpl
[], super=GridDhtTxLocalAdapter [nearOnOrig
inatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
depEnabled=false, txState=IgniteTxStateImpl [activeCac
heIds=GridLongList [idx=1, arr=[1633849959]], txMap={IgniteTxKey
[key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
hasValBytes=true], cacheId=1633849959]=IgniteTxEntry [key=KeyCacheObje
ctImpl [val=BatchIdKey [privDb=trim_sys], hasValBytes=true],
cacheId=1633849959, txKey=IgniteTxKey [key=KeyCacheObjectImpl
[val=BatchIdKey [privDb=trim_sys], hasValBytes=true], cacheId=163384995
9], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=null, filtersPassed=fal
se, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [val=BatchIdKey [privDb=trim_sys],
hasValBytes=true], val=n
ull, startVer=1473869129773, ver=GridCacheVersion [topVer=85333522,
nodeOrderDrId=10, globalTime=1473859812640, order=1473869129773],
hash=1508409679, extras=null, flags=0]]], prepared=false, lo
cked=false, nodeId=3cd37805-46a7-4287-875e-9cbd0cf03fad, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
serReadVer=null, xidVer=GridCacheVersion [topVer=853
33522, nodeOrderDrId=10, globalTime=1473859812640, order=1473869129772]]}],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=85333522,
nodeOrderDrId=10, globalTime=1473859812640, order=147
3869129772], writeVer=null, implicit=false, loc=true, threadId=50,
startTime=1473859812630, nodeId=6f7a39ba-c520-435e-9480-a42ecf0d9a58,
startVer=GridCacheVersion [topVer=85333522, nodeOrderDrId
=10, globalTime=1473859812640, order=1473869129772], endVer=null,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=5000,
sysInvalidate=false, sys=false, plc=2, commitVer=null, finaliz
ing=NONE, preparing=false, invalidParts=null, state=MARKED_ROLLBACK,
timedOut=false, topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
duration=5007ms, onePhaseCommit=false], size=1]]]
]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter$PostLockClosure2.apply(IgniteTxLocalAdapter.java:4023)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter$PostLockClosure2.apply(IgniteTxLocalAdapter.java:4010)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridEmbeddedFuture$3.applyx(GridEmbeddedFuture.java:158)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:297)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener1.apply(GridEmbeddedFuture.java:290)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:262)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:250)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:380)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:346)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.onComplete(GridDhtColocatedLockFuture.java:535)
~[ignite-core-1.5.0.final.jar:1.5.0.fi
nal]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.access$1100(GridDhtColocatedLockFuture.java:78)
~[ignite-core-1.5.0.final.jar:1.5.0.fi
nal]
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture$LockTimeoutObject.onTimeout(GridDhtColocatedLockFuture.java:1291)
~[ignite-core-1.5.0.
final.jar:1.5.0.final]
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:159)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
~[ignite-core-1.5.0.final.jar:1.5.0.final]
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7809.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@apache.org>.
It’s a topology change that impacts transactions rollback, not vice verse. I would wait while more experienced Apache committers who maintain caching components will chime in. They should be able to get to the bottom.
In the meanwhile please attach the logs from all the nodes (servers and clients, that are running on your application side). Hope you preserved them.
—
Denis
> On Sep 16, 2016, at 9:09 AM, yfernando <yo...@tudor.com> wrote:
>
> No, the node that failed was a server node.
>
> About the rollback, yes indeed. A few times that grid has hung, we have seen
> a similar lock on rollback. Why would a transaction rollback impact the
> topology?
>
> This thread dump was taken at least 10 minutes after the node died so in an
> ideal world, the grid should have recovered from the topology change which
> occurred by the node going down.
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7805.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by yfernando <yo...@tudor.com>.
No, the node that failed was a server node.
About the rollback, yes indeed. A few times that grid has hung, we have seen
a similar lock on rollback. Why would a transaction rollback impact the
topology?
This thread dump was taken at least 10 minutes after the node died so in an
ideal world, the grid should have recovered from the topology change which
occurred by the node going down.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7805.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@apache.org>.
Is the node that expected a long GC pause and failed eventually is a client node? This is important to know.
From thread dumps I see that some of the nodes unable to rollback transactions
"pub-#1%DataGridServer-Development%" Id=35 in WAITING on lock=org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture@3e4dcc0c
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:155)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
at org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.rollback(TransactionProxyImpl.java:296)
at com.somecompany.grid.server.tradegen.BatchIdHelper.getListOfIds(BatchIdHelper.java:84)
at com.somecompany.grid.server.tradegen.TradeGenerator.generateUniqueTradeId64(TradeGenerator.java:47)
at com.somecompany.grid.server.tradegen.TradeGenerator.allocateTradesFromFills(TradeGenerator.java:158)
while the others are waiting while an affinity topology changes which, in my understanding, prevents the first nodes from successful transactions rollback.
ock=org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache$AffinityReadyFuture@14a5b2c7
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:157)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.awaitTopologyVersion(GridAffinityAssignmentCache.java:477)
at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:435)
at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.primaryPartitions(GridAffinityAssignmentCache.java:399)
at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryPartitions(GridCacheAffinityManager.java:366)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.reservePartitions(GridMapQueryExecutor.java:316)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:428)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onMessage(GridMapQueryExecutor.java:184)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor$2.onMessage(GridMapQueryExecutor.java:159)
at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:1821)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103)
at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784)
at
—
Denis
> On Sep 16, 2016, at 2:44 AM, yfernando <yo...@tudor.com> wrote:
>
> Hi Denis,
>
> We have been able to reproduce this situation where a node failure freezes
> the entire grid.
>
> Please find the full thread dumps of the 5 nodes that are locked up.
>
> The memoryMode of the caches are configured to be OFFHEAP_TIERED
> The cacheMode is PARTITIONED
> The atomicityMode is TRANSACTIONAL
>
> We have also seen ALL the clients freeze during a FULL GC occurring on ANY
> single node.
>
> Please let us know if you require any more information.
>
> grid-tp1-dev-11220-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp1-dev-11220-201609141523318.txt>
> grid-tp1-dev-11223-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp1-dev-11223-201609141523318.txt>
> grid-tp3-dev-11220-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp3-dev-11220-201609141523318.txt>
> grid-tp3-dev-11221-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp3-dev-11221-201609141523318.txt>
> grid-tp4-dev-11220-201609141523318.txt
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp4-dev-11220-201609141523318.txt>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7791.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by yfernando <yo...@tudor.com>.
Hi Denis,
We have been able to reproduce this situation where a node failure freezes
the entire grid.
Please find the full thread dumps of the 5 nodes that are locked up.
The memoryMode of the caches are configured to be OFFHEAP_TIERED
The cacheMode is PARTITIONED
The atomicityMode is TRANSACTIONAL
We have also seen ALL the clients freeze during a FULL GC occurring on ANY
single node.
Please let us know if you require any more information.
grid-tp1-dev-11220-201609141523318.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp1-dev-11220-201609141523318.txt>
grid-tp1-dev-11223-201609141523318.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp1-dev-11223-201609141523318.txt>
grid-tp3-dev-11220-201609141523318.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp3-dev-11220-201609141523318.txt>
grid-tp3-dev-11221-201609141523318.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp3-dev-11221-201609141523318.txt>
grid-tp4-dev-11220-201609141523318.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n7791/grid-tp4-dev-11220-201609141523318.txt>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7791.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
Hi Binti,
See below
> On Aug 21, 2016, at 4:24 PM, bintisepaha <bi...@tudor.com> wrote:
>
> Hi Denis,
> we see this exception too from a client when the cluster is
> restarted."IllegalStateException: Cache has been closed or destroyed: cache"
> we reconnect the client to the cluster by calling Ignition.stop and
> Ignition.start/ignite again and we are able to avoid this.
>
In such a scenario there is no need to restart a client node. You need to get a new reference to a cache using Ignite.cache(…) or Ignite.getOrCreateCache(…) API. This can be done inside of a try-catch block that processes IgniteClientDisconnectedException
https://apacheignite.readme.io/docs/clients-vs-servers#client-reconnect <https://apacheignite.readme.io/docs/clients-vs-servers#client-reconnect>
In any case I think that Ignite may handle such situations automatically
https://issues.apache.org/jira/browse/IGNITE-3719 <https://issues.apache.org/jira/browse/IGNITE-3719>
> As far as cluster hanging goes, We are seeing many issues. optimistic txns
> hanging in commit(), if we try to kill nodes with hanging txns, the cluster
> hangs afterwards. unfortunately we cannot stop using Ignite at this point,
> we are already in production with some functionality.
>
> What can we send you to help us solve this issue?
Share logs and thread dumps from all the nodes.
—
Denis
>
> Thanks,
> Binti
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7199.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by bintisepaha <bi...@tudor.com>.
Hi Denis,
we see this exception too from a client when the cluster is
restarted."IllegalStateException: Cache has been closed or destroyed: cache"
we reconnect the client to the cluster by calling Ignition.stop and
Ignition.start/ignite again and we are able to avoid this.
As far as cluster hanging goes, We are seeing many issues. optimistic txns
hanging in commit(), if we try to kill nodes with hanging txns, the cluster
hangs afterwards. unfortunately we cannot stop using Ignite at this point,
we are already in production with some functionality.
What can we send you to help us solve this issue?
Thanks,
Binti
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7199.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
Hi
See below
> On Aug 22, 2016, at 8:52 AM, DLopez <d....@gmail.com> wrote:
>
> Hi Denis,
> The exception was happening to us in a cluster where all the nodes have the same configuration, they use the same piece of code to setup the cluster, and happened without restarting any node at all.
Please refer to my latest answer to Binti regarding this exception.
> Regarding the first issue, one node being slow due to GC issues would slow down the whole cluster even if just reading cache values.
>
> Regarding our solution, we really just needed a set of replicated caches with periodic updates and just 1-2 caches updated more frequently but we didn't need up-to-the-ms updates or grid capability, so we created our own custom solution skipping most of the problems that make this kind of sofware hard and we did not really need. In fact, Ignite was the 6th product that we discarded after implementation, with a couple more discareded even sooner. It seems the industry is now more focused on the grid part so we have been unable to find a good fit for our use case.
>
Got you. It’s pity. Hope that we will get to the bottom of the issue with the help from Binti side.
—
Denis
> D.
>
> 2016-08-21 7:42 GMT+02:00 Denis Magda [via Apache Ignite Users] <[hidden email] <x-msg://7/user/SendEmail.jtp?type=node&node=7211&i=0>>:
> This exception usually happens when a cluster, where a cache had been initially deployed, was restarted without the cache recreation and after a reconnection to the cluster a client node tries to access the cache that is not longer exists. This exception can be easily avoided if all the caches are defined in server nodes configuration, that is used during startup, or if a client creates a cache one more time after the reconnection.
>
> BTW, what kind of IMDG did you switch to?
>
> —
> Denis
>
>> On Aug 20, 2016, at 1:28 AM, DLopez <[hidden email] <http://user/SendEmail.jtp?type=node&node=7192&i=0>> wrote:
>>
>> Hi Binti,
>> No, we were unable to resolve it. We were also experiencing quite frequently from another known issue "IllegalStateException: Cache has been closed or destroyed: cache" so we decided to move on and drop Ignite altogether. We finished the transition in production a couple of weeks ago.
>> I'm sorry I can't be of much help,
>> D.
>>
>> 2016-08-20 1:31 GMT+02:00 bintisepaha [via Apache Ignite Users] <<a href="x-msg://7/user/ <x-msg://7/user/SendEmail.jtp?type=node&node=7188&i=0>SendEmail.jtp?type=node& <x-msg://7/user/SendEmail.jtp?type=node&node=7188&i=0>node=7188&i=0 <x-msg://7/user/SendEmail.jtp?type=node&node=7188&i=0>" target="_top" rel="nofollow" link="external" class="">[hidden email]>:
>> Hi, we are seeing similar issues too with ignite 1.5.0. Were you guys able to resolve it? we use distributed caches in full-sync mode and also a few replicated caches.
>>
>> Thanks,
>> Binti
>>
>> If you reply to this email, your message will be added to the discussion below:
>> http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7183.html <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7183.html>
>> To unsubscribe from One failing node stalling the whole cluster, click here <>.
>> NAML <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>> View this message in context: Re: One failing node stalling the whole cluster <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7188.html>
>> Sent from the Apache Ignite Users mailing list archive <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com <http://nabble.com/>.
>
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7192.html <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7192.html>
> To unsubscribe from One failing node stalling the whole cluster, click here <applewebdata://07AF829C-CBE7-47E1-8877-3CA124A6D355>.
> NAML <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: One failing node stalling the whole cluster <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7211.html>
> Sent from the Apache Ignite Users mailing list archive <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com <http://nabble.com/>.
Re: One failing node stalling the whole cluster
Posted by DLopez <d....@gmail.com>.
Hi Denis,
The exception was happening to us in a cluster where all the nodes have the
same configuration, they use the same piece of code to setup the cluster,
and happened without restarting any node at all. Regarding the first issue,
one node being slow due to GC issues would slow down the whole cluster even
if just reading cache values.
Regarding our solution, we really just needed a set of replicated caches
with periodic updates and just 1-2 caches updated more frequently but we
didn't need up-to-the-ms updates or grid capability, so we created our own
custom solution skipping most of the problems that make this kind of
sofware hard and we did not really need. In fact, Ignite was the 6th
product that we discarded after implementation, with a couple more
discareded even sooner. It seems the industry is now more focused on the
grid part so we have been unable to find a good fit for our use case.
D.
2016-08-21 7:42 GMT+02:00 Denis Magda [via Apache Ignite Users] <
ml-node+s70518n7192h72@n6.nabble.com>:
> This exception usually happens when a cluster, where a cache had been
> initially deployed, was restarted without the cache recreation and after a
> reconnection to the cluster a client node tries to access the cache that is
> not longer exists. This exception can be easily avoided if all the caches
> are defined in server nodes configuration, that is used during startup, or
> if a client creates a cache one more time after the reconnection.
>
> BTW, what kind of IMDG did you switch to?
>
> —
> Denis
>
>
> On Aug 20, 2016, at 1:28 AM, DLopez <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=7192&i=0>> wrote:
>
> Hi Binti,
> No, we were unable to resolve it. We were also experiencing quite
> frequently from another known issue "IllegalStateException: Cache has been
> closed or destroyed: cache" so we decided to move on and drop Ignite
> altogether. We finished the transition in production a couple of weeks ago.
> I'm sorry I can't be of much help,
> D.
>
> 2016-08-20 1:31 GMT+02:00 bintisepaha [via Apache Ignite Users] <<a
> href="x-msg://7/user/SendEmail.jtp?type=node&node=7188&i=0"
> target="_top" rel="nofollow" link="external" class="">[hidden email]>:
>
>> Hi, we are seeing similar issues too with ignite 1.5.0. Were you guys
>> able to resolve it? we use distributed caches in full-sync mode and also a
>> few replicated caches.
>>
>> Thanks,
>> Binti
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-ignite-users.70518.x6.nabble.com/One-failing-
>> node-stalling-the-whole-cluster-tp5372p7183.html
>> To unsubscribe from One failing node stalling the whole cluster, click
>> here.
>> NAML
>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: One failing node stalling the whole
> cluster
> <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7188.html>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com
> <http://nabble.com>.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/One-
> failing-node-stalling-the-whole-cluster-tp5372p7192.html
> To unsubscribe from One failing node stalling the whole cluster, click
> here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5372&code=ZC5sb3Blei5qQGdtYWlsLmNvbXw1MzcyfDIwNTkzNjQ3OTE=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7211.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
This exception usually happens when a cluster, where a cache had been initially deployed, was restarted without the cache recreation and after a reconnection to the cluster a client node tries to access the cache that is not longer exists. This exception can be easily avoided if all the caches are defined in server nodes configuration, that is used during startup, or if a client creates a cache one more time after the reconnection.
BTW, what kind of IMDG did you switch to?
—
Denis
> On Aug 20, 2016, at 1:28 AM, DLopez <d....@gmail.com> wrote:
>
> Hi Binti,
> No, we were unable to resolve it. We were also experiencing quite frequently from another known issue "IllegalStateException: Cache has been closed or destroyed: cache" so we decided to move on and drop Ignite altogether. We finished the transition in production a couple of weeks ago.
> I'm sorry I can't be of much help,
> D.
>
> 2016-08-20 1:31 GMT+02:00 bintisepaha [via Apache Ignite Users] <[hidden email] <x-msg://7/user/SendEmail.jtp?type=node&node=7188&i=0>>:
> Hi, we are seeing similar issues too with ignite 1.5.0. Were you guys able to resolve it? we use distributed caches in full-sync mode and also a few replicated caches.
>
> Thanks,
> Binti
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7183.html <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7183.html>
> To unsubscribe from One failing node stalling the whole cluster, click here <applewebdata://9CA93834-A5E4-4CB1-95AE-E562768298E2>.
> NAML <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: One failing node stalling the whole cluster <http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7188.html>
> Sent from the Apache Ignite Users mailing list archive <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by DLopez <d....@gmail.com>.
Hi Binti,
No, we were unable to resolve it. We were also experiencing quite
frequently from another known issue "IllegalStateException: Cache has been
closed or destroyed: cache" so we decided to move on and drop Ignite
altogether. We finished the transition in production a couple of weeks ago.
I'm sorry I can't be of much help,
D.
2016-08-20 1:31 GMT+02:00 bintisepaha [via Apache Ignite Users] <
ml-node+s70518n7183h93@n6.nabble.com>:
> Hi, we are seeing similar issues too with ignite 1.5.0. Were you guys able
> to resolve it? we use distributed caches in full-sync mode and also a few
> replicated caches.
>
> Thanks,
> Binti
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/One-
> failing-node-stalling-the-whole-cluster-tp5372p7183.html
> To unsubscribe from One failing node stalling the whole cluster, click
> here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5372&code=ZC5sb3Blei5qQGdtYWlsLmNvbXw1MzcyfDIwNTkzNjQ3OTE=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7188.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by bintisepaha <bi...@tudor.com>.
Hi, we are seeing similar issues too with ignite 1.5.0. Were you guys able to
resolve it? we use distributed caches in full-sync mode and also a few
replicated caches.
Thanks,
Binti
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p7183.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Kristian Rosenvold <kr...@apache.org>.
We're also seeing this total hang of our replicated cache cluster when a
single node goes totally lethargic due to too heavy memory load. The
culprit node typically does not respond to "jstack" due to either excessive
memory load or missing safepoints. Sometimes we need to do kill -9 to get
the node down.
I have been planning to do a jstack on the *remaining* nodes to try to
figure out why they appear to not be timing out the non-responsive node. I
will upgrade to 1.6 and see if I can pinpoint the problem there.
Kristian
2016-06-05 21:18 GMT+02:00 DLopez <d....@gmail.com>:
> Hi Dennis,
> I agree that it shouldn't happen but I have been able to reproduce it in
> other machines consistently and the only "connection" that they have is
> that
> they share the Ignite replicated caches.
>
> One machine is basically reading from several caches and filling up some
> data to be returned, I can have 25 clients requesting some data and
> everything is fine. The other one is a different application, that
> basically
> fills up the replicated caches from the DB but receives no direct requests.
> Someone forgot to control a batch job in this second application and it can
> be run many times, consuming up all the memory in this second application.
> The strange thing is that when the second applications start GCing like
> crazy, the first one starts going slower and slower up to a point when it
> stops answering requests. If I kill -9 the second application, the first
> one
> goes back to normal behaviour immediately and can respond 25 simultaneous
> requests again with normal response times. I can restart the second
> application and repeat the same thing and the behaviour is the same.
>
> So I can tell you it's no application code or garbage collection issue in
> the other app. The batch job in the second app, that we run manually for
> this test, is not replicated and does nothing related to ignite, it does
> not
> even use the replicated caches.
>
> The only thing I can think of that would show this behaviour would be the
> sync. process in the Ignite caches slowing down/stalling the reading of
> values. As the second app. starts experiencing GC issues and slows down the
> Ignite sync. process, then it affects the other apps reading the caches. So
> I was wondering if the sync. mechanism might have some kind of lock on the
> caches that would prevent reading from them.
>
> I'll see if I can replicate it in a small scale experiment, apart from
> testing with ignite 1.6.
>
> Thanks for your input
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p5432.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: One failing node stalling the whole cluster
Posted by DLopez <d....@gmail.com>.
Hi Dennis,
I agree that it shouldn't happen but I have been able to reproduce it in
other machines consistently and the only "connection" that they have is that
they share the Ignite replicated caches.
One machine is basically reading from several caches and filling up some
data to be returned, I can have 25 clients requesting some data and
everything is fine. The other one is a different application, that basically
fills up the replicated caches from the DB but receives no direct requests.
Someone forgot to control a batch job in this second application and it can
be run many times, consuming up all the memory in this second application.
The strange thing is that when the second applications start GCing like
crazy, the first one starts going slower and slower up to a point when it
stops answering requests. If I kill -9 the second application, the first one
goes back to normal behaviour immediately and can respond 25 simultaneous
requests again with normal response times. I can restart the second
application and repeat the same thing and the behaviour is the same.
So I can tell you it's no application code or garbage collection issue in
the other app. The batch job in the second app, that we run manually for
this test, is not replicated and does nothing related to ignite, it does not
even use the replicated caches.
The only thing I can think of that would show this behaviour would be the
sync. process in the Ignite caches slowing down/stalling the reading of
values. As the second app. starts experiencing GC issues and slows down the
Ignite sync. process, then it affects the other apps reading the caches. So
I was wondering if the sync. mechanism might have some kind of lock on the
caches that would prevent reading from them.
I'll see if I can replicate it in a small scale experiment, apart from
testing with ignite 1.6.
Thanks for your input
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/One-failing-node-stalling-the-whole-cluster-tp5372p5432.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: One failing node stalling the whole cluster
Posted by Denis Magda <dm...@gridgain.com>.
Hi Daniel,
Actually a failure of one machine shouldn’t lead to the whole cluster shutdown unless your application code was executed on those nodes as well and killed other nodes due to high GC pauses or some other reason.
My first suggestion is to tune garbage collection appropriately:
https://apacheignite.readme.io/v1.6/docs/jvm-and-system-tuning#jvm-tuning-for-clusters-with-on_heap-caches
and track GC logs to adjust the settings if needed
https://apacheignite.readme.io/v1.6/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats
If the issue still happens please share GC logs and logs from all the nodes with us. Probably we will be able to pin point the problem on your side.
—
Denis
> On Jun 2, 2016, at 11:21 AM, Daniel López <d....@gmail.com> wrote:
>
> Hi there,
>
> We are using Ignite 1.5.0 and we are experiencing a strange issue where one node stalls the others node in the cluster. We are using CacheMode.REPLICATED caches to store data on heap on several nodes to improve latency.
> In one of the latest upgrades someone introduced a bug in the system that could cause one node to consume too much memory and start having GC issues. Sh*t happens :).
> The problem, however, is that when this node starts to crawl due to heavy GC usage, it starts spitting these logs:
>
> |Failed to process selector key (will close): GridSelectorNioSessionImpl [selectorIdx=0, queueSize=24, writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], recovery=GridNioRecoveryDescriptor [acked=284144, resendCnt=0, rcvCnt=284230, reserved=true, lastAck=284224, nodeLeft=false, node=TcpDiscoveryNode [id=1109a421-ec72-4534-99c4-df5d7e4f6136, addrs=[x.y.z4, 127.0.0.1], sockAddrs=[machine.env/x.y.z4:3808, /x.y.z:3808, /127.0.0.1:3808 <http://127.0.0.1:3808/>], discPort=3808, order=32, intOrder=17, lastExchangeTime=1464854224616, loc=false, ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], connected=true, connectCnt=0, queueLimit=5120], super=GridNioSessionImpl [locAddr=/x.y.z3:47100, rmtAddr=/x.y.z4:33450, createTime=1464854224707, closeTime=0, bytesSent=10838655, bytesRcvd=221207982, sndSchedTime=1464854575433, lastSndTime=1464854575433, lastRcvTime=1464854575433, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@63f44e30, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]]
> WARN |2016-06-02T08:03:02,575||TcpCommunicationSpi|Closing NIO session because of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, msg=Conexión reinicializada por la máquina remota]
>
> And the other nodes in the cluster start to produce these other logs and access to the cache slows down/pauses greatly:
>
> GridCachePartitionExchangeManager|Failed to send partitions full message [node=TcpDiscoveryNode [id=913ea465-ed45-4ec9-a4b7-d2c5f9c57a2e, a....
> TcpDiscoverySpi|Failed to ping node (status check will be initiated): ....
> GridDiscoveryManager|Node FAILED: TcpDiscoveryNode [id=...
>
> That the node with the GC issues stops working is "normal", even if undesired, but what really worries us is that it causes the other nodes in the cluster to stop being able to use the replicated caches, so one node can bring down the whole cluster.
>
> If we stop the offeding node, the others go back to normal behaviour and work as fast as always.
>
> We are going to solve the application bug, of course, but is there any configuration setting that we can tweak so one bug in one machine does not bring the whole cluster to a halt?
>
> Ignite is configured to use TcpDiscoverySpi with TcpDiscoveryVmIpFinder with a list of addresses (11 nodes per set currently)
> Each node has 29 caches configured like this:
> cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
> cacheConfiguration.setCopyOnRead(false);
> cacheConfiguration.setEagerTtl(false);
>
> Thanks,
> D.
>
> PD: Yes, we'll have to try with latest Ignite version but we wanted to know if there is any configuration setting that might help first, before having to migrate and restart the whole testing/process.