You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ght230 <gh...@163.com> on 2017/03/22 12:01:25 UTC
[GridCachePartitionExchangeManager] Pending transaction deadlock
detection futures
I start 4 Ignite servers, then I start ignitevisorcmd to view the cache.
But sometimes the ignitevisorcmd will stop at
"Security status [authentication=off, tls/ssl=off]"
and it will stuck the whole cluster.
From the log of ignitevisorcmd, we can find the following info.
"[16:52:23,640][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
exchange futures:
[16:52:23,641][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Last 10
exchange futures (total: 1):
[16:52:23,646][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] >>>
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=c0256629-1950-45cf-98cb-e1d874e907a0, nodeName=null, addrs=[10.10.10.81,
127.0.0.1, 192.168.14.81], sockAddrs=[/127.0.0.1:47504, /10.10.10.81:47504,
dn81.dcs.com/192.168.14.81:47504], discPort=47504, order=8, intOrder=7,
lastExchangeTime=1490172142809, loc=true, ver=1.6.12#19700101-sha1:00000000,
isClient=false], topVer=8, nodeId8=c0256629, msg=null, type=NODE_JOINED,
tstamp=1490172143132], crd=TcpDiscoveryNode
[id=b0b8409c-36de-42c2-92c2-65f583941e07, nodeName=null, addrs=[10.10.10.81,
127.0.0.1, 192.168.14.81], sockAddrs=[dn81.dcs.com/192.168.14.81:47500,
/127.0.0.1:47500, /10.10.10.81:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1490172143021, loc=false,
ver=1.6.12#19700101-sha1:00000000, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=8,
minorTopVer=0], nodeId=c0256629, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490172143567,
endTime=1490172143617, ignoreInterrupts=false, state=DONE], init=true,
topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=null,
skipPreload=true, clientOnlyExchange=true, initTs=1490172143567,
centralizedAff=false, forcedRebFut=null, evtLatch=0,
remaining=[b0b8409c-36de-42c2-92c2-65f583941e07,
d55c68e1-ddbd-435b-9c74-f7b59912cd49, b208aa0c-08bd-4f93-9c59-ac5ea49bdfda,
6fdfdedc-d797-4c62-902e-26df6aa97b57], srvNodes=[TcpDiscoveryNode
[id=b0b8409c-36de-42c2-92c2-65f583941e07, nodeName=null, addrs=[10.10.10.81,
127.0.0.1, 192.168.14.81], sockAddrs=[dn81.dcs.com/192.168.14.81:47500,
/127.0.0.1:47500, /10.10.10.81:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1490172143021, loc=false,
ver=1.6.12#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode
[id=d55c68e1-ddbd-435b-9c74-f7b59912cd49, nodeName=null, addrs=[10.10.10.81,
127.0.0.1, 192.168.14.81], sockAddrs=[/10.10.10.81:47501, /127.0.0.1:47501,
dn81.dcs.com/192.168.14.81:47501], discPort=47501, order=2, intOrder=2,
lastExchangeTime=1490172143031, loc=false,
ver=1.6.12#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode
[id=6fdfdedc-d797-4c62-902e-26df6aa97b57, nodeName=null, addrs=[10.10.10.81,
127.0.0.1, 192.168.14.81], sockAddrs=[/10.10.10.81:47502,
dn81.dcs.com/192.168.14.81:47502, /127.0.0.1:47502], discPort=47502,
order=4, intOrder=4, lastExchangeTime=1490172143041, loc=false,
ver=1.6.12#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode
[id=b208aa0c-08bd-4f93-9c59-ac5ea49bdfda, nodeName=null, addrs=[10.10.10.81,
127.0.0.1, 192.168.14.81], sockAddrs=[/127.0.0.1:47503, /10.10.10.81:47503,
dn81.dcs.com/192.168.14.81:47503], discPort=47503, order=5, intOrder=5,
lastExchangeTime=1490172143051, loc=false,
ver=1.6.12#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter
[resFlag=0, res=null, startTime=1490172143567, endTime=0,
ignoreInterrupts=false, state=INIT]]
[16:52:23,650][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
transactions:
[16:52:23,651][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
explicit locks:
[16:52:23,651][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
cache futures:
[16:52:23,652][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
atomic cache futures:
[16:52:23,653][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
data streamer futures:
[16:52:23,654][WARN
][exchange-worker-#300%null%][GridCachePartitionExchangeManager] Pending
transaction deadlock detection futures:
[16:52:23,655][WARN ][exchange-worker-#300%null%][TcpCommunicationSpi]
Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=b0b8409c-36de-42c2-92c2-65f583941e07, idx=0,
connCnt=-1], msgsSent=1, msgsAckedByRmt=1, msgsRcvd=0, lastAcked=0,
reserveCnt=1, descIdHash=1052107904]
Communication SPI clients: "
Does anyone know about it?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: [GridCachePartitionExchangeManager] Pending transaction
deadlock detection futures
Posted by ght230 <gh...@163.com>.
I have created a JIRA for that.
https://issues.apache.org/jira/browse/IGNITE-4924.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362p11767.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: [GridCachePartitionExchangeManager] Pending transaction deadlock
detection futures
Posted by Alexey Kuznetsov <ak...@apache.org>.
Hi ght230,
I think that Visor CMD should work in client mode, but it is not
implemented yet.
Please create issue in Ignite JIRA for that.
On Sat, Apr 1, 2017 at 1:41 PM, ght230 <gh...@163.com> wrote:
> I want to know should Visor CMD be worked in client mode or daemon mode?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-
> deadlock-detection-futures-tp11362p11638.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Alexey Kuznetsov
Re: [GridCachePartitionExchangeManager] Pending transaction
deadlock detection futures
Posted by ght230 <gh...@163.com>.
I want to know should Visor CMD be worked in client mode or daemon mode?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362p11638.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: [GridCachePartitionExchangeManager] Pending transaction deadlock
detection futures
Posted by Alexey Kuznetsov <ak...@apache.org>.
Val,
>> Does anyone have an idea why client mode in Visor affects behavior? I
thought we already forced client mode there, no?
Visor CMD was NOT reworked to client mode.
So Visor CMD starts server node in daemon mode.
On Wed, Mar 29, 2017 at 2:28 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> cross-posting to dev list.
>
> Guys,
>
> Does anyone have an idea why client mode in Visor affects behavior? I
> thought we already forced client mode there, no?
>
> Alexey, you should know the answer. Can you please take a look at this
> thread?
>
> -Val
>
>
> On Tue, Mar 28, 2017 at 7:02 AM, ght230 <gh...@163.com> wrote:
>
>> Yes,I use custom build.
>>
>> Today I tried to set Ignition.setClientMode(true) for command "open" of
>> the
>> visorcmd.
>>
>> It seems the visorcmd will not stuck the whole cluster again.
>>
>> Is there anything wrong in "ClientMode" of the visorcmd?
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/GridCachePartitionExchangeManager-Pending-t
>> ransaction-deadlock-detection-futures-tp11362p11501.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
--
Alexey Kuznetsov
Re: [GridCachePartitionExchangeManager] Pending transaction deadlock
detection futures
Posted by Alexey Kuznetsov <ak...@apache.org>.
Val,
>> Does anyone have an idea why client mode in Visor affects behavior? I
thought we already forced client mode there, no?
Visor CMD was NOT reworked to client mode.
So Visor CMD starts server node in daemon mode.
On Wed, Mar 29, 2017 at 2:28 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> cross-posting to dev list.
>
> Guys,
>
> Does anyone have an idea why client mode in Visor affects behavior? I
> thought we already forced client mode there, no?
>
> Alexey, you should know the answer. Can you please take a look at this
> thread?
>
> -Val
>
>
> On Tue, Mar 28, 2017 at 7:02 AM, ght230 <gh...@163.com> wrote:
>
>> Yes,I use custom build.
>>
>> Today I tried to set Ignition.setClientMode(true) for command "open" of
>> the
>> visorcmd.
>>
>> It seems the visorcmd will not stuck the whole cluster again.
>>
>> Is there anything wrong in "ClientMode" of the visorcmd?
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/GridCachePartitionExchangeManager-Pending-t
>> ransaction-deadlock-detection-futures-tp11362p11501.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
--
Alexey Kuznetsov
Re: [GridCachePartitionExchangeManager] Pending transaction deadlock
detection futures
Posted by Valentin Kulichenko <va...@gmail.com>.
cross-posting to dev list.
Guys,
Does anyone have an idea why client mode in Visor affects behavior? I
thought we already forced client mode there, no?
Alexey, you should know the answer. Can you please take a look at this
thread?
-Val
On Tue, Mar 28, 2017 at 7:02 AM, ght230 <gh...@163.com> wrote:
> Yes,I use custom build.
>
> Today I tried to set Ignition.setClientMode(true) for command "open" of the
> visorcmd.
>
> It seems the visorcmd will not stuck the whole cluster again.
>
> Is there anything wrong in "ClientMode" of the visorcmd?
>
>
>
> --
> View this message in context: http://apache-ignite-users.705
> 18.x6.nabble.com/GridCachePartitionExchangeManager-Pending-
> transaction-deadlock-detection-futures-tp11362p11501.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: [GridCachePartitionExchangeManager] Pending transaction deadlock
detection futures
Posted by Valentin Kulichenko <va...@gmail.com>.
cross-posting to dev list.
Guys,
Does anyone have an idea why client mode in Visor affects behavior? I
thought we already forced client mode there, no?
Alexey, you should know the answer. Can you please take a look at this
thread?
-Val
On Tue, Mar 28, 2017 at 7:02 AM, ght230 <gh...@163.com> wrote:
> Yes,I use custom build.
>
> Today I tried to set Ignition.setClientMode(true) for command "open" of the
> visorcmd.
>
> It seems the visorcmd will not stuck the whole cluster again.
>
> Is there anything wrong in "ClientMode" of the visorcmd?
>
>
>
> --
> View this message in context: http://apache-ignite-users.705
> 18.x6.nabble.com/GridCachePartitionExchangeManager-Pending-
> transaction-deadlock-detection-futures-tp11362p11501.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: [GridCachePartitionExchangeManager] Pending transaction
deadlock detection futures
Posted by ght230 <gh...@163.com>.
Yes,I use custom build.
Today I tried to set Ignition.setClientMode(true) for command "open" of the
visorcmd.
It seems the visorcmd will not stuck the whole cluster again.
Is there anything wrong in "ClientMode" of the visorcmd?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362p11501.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: [GridCachePartitionExchangeManager] Pending transaction
deadlock detection futures
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
Do you use custom build? In logs Ignite version 1.6.12, but it uses stripped
pool, which was introduced only in 1.8.2 and 1.9.
Also there was a node stall, because of many disk operations (swap file).
This doesn't lead to hang, but may reduce cache performance.
Besides log files also thread dumps required, because they help to find out
the reason of the hang. Please, attach them as well.
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362p11485.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: [GridCachePartitionExchangeManager] Pending transaction
deadlock detection futures
Posted by ght230 <gh...@163.com>.
send.rar
<http://apache-ignite-users.70518.x6.nabble.com/file/n11455/send.rar>
Yesterday, I tested it again.This time I do not start visorcmd.
I started 2 servers and 1 client at 21:26(client will receive message from
MQ,server will save this message and process it).
The speed of receiving message is 190/s.
After 12 hours, I start another 2 servers.
About 1 hour later, the cluster stucked.
The attachment is the log of all nodes(4 servers and 1 client) in 2
machines.
Please help me to confirm them.
Thank you!
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362p11455.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: [GridCachePartitionExchangeManager] Pending transaction
deadlock detection futures
Posted by dkarachentsev <dk...@gridgain.com>.
Hi,
What operations in visorcmd you did?
Please attach thread dumps (when cluster hangs) and logs from all nodes.
Thanks!
-Dmitry.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/GridCachePartitionExchangeManager-Pending-transaction-deadlock-detection-futures-tp11362p11386.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.