Apache Ignite 2.10 | Ignite Cluster Crashed

["siva.velicheti.barclays.com via user" <us...@ignite.apache.org>]

Hi,

We have Ignite Cluster with 6 nodes. We have seen a sudden crash of all 6 nodes at the same time with below error. Attached full log from one of the node.

2024-05-19 00:12:21,923 ERROR [disco-event-worker-#61%app-cache%] o.a.i.i.p.f.FailureProcessor [JavaLogger.java:300] Ignite node is in invalid state due to a critical failure.
2024-05-19 00:12:21,923 WARN [disco-event-worker-#61%app-cache%] o.a.i.i.m.d.GridDiscoveryManager [JavaLogger.java:295] Node FAILED: TcpDiscoveryNode [id=969a3585-4a17-4442-9d82-effb1c4b5c29, consistentId=*********, addrs=ArrayList [10.141.1.223, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, *********/10.141.1.223:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1715709201672, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false]
2024-05-19 00:12:21,923 ERROR [node-stopper]  [LogManager.java:1650] Stopping local node on Ignite failure: [failureCtx=FailureContext [type=SEGMENTATION, err=null]]

Few recent changes we did, include  - increasing the cluster size from 3 to 6. We have deployed compute jobs which run on call 6 nodes. Jobs run every 10 mins. Appreciate any help on this.

Thanks,
Siva


This message is for information purposes only. It is not a recommendation, advice, offer or solicitation to buy or sell a product or service, nor an official confirmation of any transaction. It is directed at persons who are professionals and is intended for the recipient(s) only. It is not directed at retail customers. This message is subject to the terms at: https://www.ib.barclays/disclosures/web-and-email-disclaimer.html. 

For important disclosures, please see: https://www.ib.barclays/disclosures/sales-and-trading-disclaimer.html regarding marketing commentary from Barclays Sales and/or Trading desks, who are active market participants; https://www.ib.barclays/disclosures/barclays-global-markets-disclosures.html regarding our standard terms for Barclays Investment Bank where we trade with you in principal-to-principal wholesale markets transactions; and in respect to Barclays Research, including disclosures relating to specific issuers, see: http://publicresearch.barclays.com.
__________________________________________________________________________________ 
If you are incorporated or operating in Australia, read these important disclosures: https://www.ib.barclays/disclosures/important-disclosures-asia-pacific.html.
__________________________________________________________________________________
For more details about how we use personal information, see our privacy notice: https://www.ib.barclays/disclosures/personal-information-use.html. 
__________________________________________________________________________________