You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Kurt Greaves (JIRA)" <ji...@apache.org> on 2018/06/25 11:53:00 UTC
[jira] [Resolved] (CASSANDRA-14470) Repair validation failed/unable
to create merkle tree
[ https://issues.apache.org/jira/browse/CASSANDRA-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kurt Greaves resolved CASSANDRA-14470.
--------------------------------------
Resolution: Duplicate
> Repair validation failed/unable to create merkle tree
> -----------------------------------------------------
>
> Key: CASSANDRA-14470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14470
> Project: Cassandra
> Issue Type: Bug
> Reporter: Harry Hough
> Priority: Major
>
> I had trouble repairing with a full repair across all nodes and keyspaces so I swapped to doing table by table. This table will not repair even after scrub/restart of all nodes. I am using command:
> {code:java}
> nodetool repair -full -seq keyspace table
> {code}
> {code:java}
> [2018-05-25 19:26:36,525] Repair session 0198ee50-6050-11e8-a3b7-9d0793eab507 for range [(165598500763544933,166800441975877433], (-5455068259072262254,-5445777107512274819], (-4614366950466274594,-4609359222424798148], (3417371506258365094,3421921915575816226], (5221788898381458942,5222846663270250559], (3421921915575816226,3429175540277204991], (3276484330153091115,3282213186258578546], (-3306169730424140596,-3303439264231406101], (5228704360821395206,5242415853745535023], (5808045095951939338,5808562658315740708], (-3303439264231406101,-3302592736123212969]] finished (progress: 1%)
> [2018-05-25 19:27:23,848] Repair session 0180f980-6050-11e8-a3b7-9d0793eab507 for range [(-8495158945319933291,-8482949618583319581], (1803296697741516342,1805330812863783941], (8633191319643427141,8637771071728131257], (2214097236323810344,2218253238829661319], (8637771071728131257,8639627594735133685], (2195525904029414718,2214097236323810344], (-8500127431270773970,-8495158945319933291], (7151693083782264341,7152162989417914407], (-8482949618583319581,-8481973749935314249]] finished (progress: 1%)
> [2018-05-25 19:30:32,590] Repair session 01ac9d62-6050-11e8-a3b7-9d0793eab507 for range [(7887346492105510731,7893062759268864220], (-153277717939330979,-151986584968539220], (-6351665356961460262,-6336288442758847669], (7881942012672602731,7887346492105510731], (-5884528383037906783,-5878097817437987368], (6054625594262089428,6060773114960761336], (-6354401100436622515,-6351665356961460262], (3358411934943460772,3363367777663817876], (6255644242745576360,6278718135193665575], (-6321106762570843270,-6316788220143151823], (1754319239259058661,1759314644652031521], (7893062759268864220,7894890594190784729], (-8012293411840276426,-8011781808288431224]] failed with error [repair #01ac9d62-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(7887346492105510731,7893062759268864220], (-153277717939330979,-151986584968539220], (-6351665356961460262,-6336288442758847669], (7881942012672602731,7887346492105510731],
> (-5884528383037906783,-5878097817437987368], (6054625594262089428,6060773114960761336], (-6354401100436622515,-6351665356961460262], (3358411934943460772,3363367777663817876], (6255644242745576360,6278718135193665575], (-6321106762570843270,-6316788220143151823], (1754319239259058661,1759314644652031521], (7893062759268864220,7894890594190784729], (-8012293411840276426,-8011781808288431224]]] Validation failed in /192.168.8.64 (progress: 1%)
> [2018-05-25 19:30:38,744] Repair session 01ab16c1-6050-11e8-a3b7-9d0793eab507 for range [(4474598255414218354,4477186372547790770], (-8368931070988054567,-8367389908801757978], (4445104759712094068,4445123832517144036], (6749641233379918040,6749879473217708908], (717627050679001698,729408043324000761], (8984622403893999385,8990662643404904110], (4457612694557846994,4474598255414218354], (5589049422573545528,5593079877787783784], (3609693317839644945,3613727999875360405], (8499016262183246473,8504603366117127178], (-5421277973540712245,-5417725796037372830], (5586405751301680690,5589049422573545528], (-2611069890590917549,-2603911539353128123], (2424772330724108233,2427564448454334730], (3172651438220766183,3175226710613527829], (4445123832517144036,4457612694557846994], (-6827531712183440570,-6800863837312326365], (5593079877787783784,5596020904874304252], (716705770783505310,717627050679001698], (115377252345874298,119626359210683992], (239394377432130766,240250561347730054]] failed with error [repair #01ab16c1-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(4474598255414218354,4477186372547790770], (-8368931070988054567,-8367389908801757978], (4445104759712094068,4445123832517144036], (6749641233379918040,6749879473217708908], (717627050679001698,729408043324000761], (8984622403893999385,8990662643404904110], (4457612694557846994,4474598255414218354], (5589049422573545528,5593079877787783784], (3609693317839644945,3613727999875360405], (8499016262183246473,8504603366117127178], (-5421277973540712245,-5417725796037372830], (5586405751301680690,5589049422573545528], (-2611069890590917549,-2603911539353128123], (2424772330724108233,2427564448454334730], (3172651438220766183,3175226710613527829], (4445123832517144036,4457612694557846994], (-6827531712183440570,-6800863837312326365], (5593079877787783784,5596020904874304252], (716705770783505310,717627050679001698], (115377252345874298,119626359210683992], (239394377432130766,240250561347730054]]] Validation failed in
> /192.168.8.63 (progress: 1%)
> [2018-05-25 19:31:49,787] Repair session 01a4ae20-6050-11e8-a3b7-9d0793eab507 for range [(-2541759376733803975,-2534654569942446346], (5879245607426320709,5880012885546321040], (-6369551868880447648,-6359409984081717656], (-6599114937188060013,-6597469275333616279], (-5074096572632539578,-5067488659471711472], (-6379754598016153113,-6369551868880447648], (2064405355459946002,2071996664850745669], (-2534654569942446346,-2517719430302560572], (7881309182913674059,7881942012672602731], (-2544088936726049385,-2541759376733803975], (2279496339605311864,2281121064700207175], (7872992433920056063,7881309182913674059], (2062114659748646544,2064405355459946002], (-2150878401005443227,-2148033787477253835], (-1741268532521628862,-1723492194304925672], (-2148033787477253835,-2148008030576152684], (2274175180327961853,2279496339605311864]] failed with error [repair #01a4ae20-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(-2541759376733803975,-2534654569942446346], (5879245607426320709,5880012885546321040], (-6369551868880447648,-6359409984081717656], (-6599114937188060013,-6597469275333616279], (-5074096572632539578,-5067488659471711472], (-6379754598016153113,-6369551868880447648], (2064405355459946002,2071996664850745669], (-2534654569942446346,-2517719430302560572], (7881309182913674059,7881942012672602731], (-2544088936726049385,-2541759376733803975], (2279496339605311864,2281121064700207175], (7872992433920056063,7881309182913674059], (2062114659748646544,2064405355459946002], (-2150878401005443227,-2148033787477253835], (-1741268532521628862,-1723492194304925672], (-2148033787477253835,-2148008030576152684], (2274175180327961853,2279496339605311864]]] Validation failed in /192.168.8.64 (progress: 1%)
> [2018-05-25 19:31:49,845] Repair session 01c26f52-6050-11e8-a3b7-9d0793eab507 for range [(-6336288442758847669,-6327494039552357362], (-6596499651894591521,-6570651311582753946], (-6597469275333616279,-6596499651894591521], (2057770067222008303,2062114659748646544], (-5870054111151365631,-5835304364517776345], (-3812151910311844467,-3802006636037441627], (-2619800330042834297,-2615481117037091603], (4808940926778034213,4810350864294758856], (-7508256920307222829,-7506372018227268626], (-7104590653728972577,-7104546570237712729], (3158009800098518496,3172651438220766183], (-2615481117037091603,-2611069890590917549], (-5878097817437987368,-5870054111151365631], (-2547658065527858190,-2544088936726049385], (232652608016417486,239394377432130766], (3154311195118940026,3158009800098518496]] failed with error [repair #01c26f52-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(-6336288442758847669,-6327494039552357362], (-6596499651894591521,-6570651311582753946], (-6597469275333616279,-6596499651894591521], (2057770067222008303,2062114659748646544], (-5870054111151365631,-5835304364517776345], (-3812151910311844467,-3802006636037441627], (-2619800330042834297,-2615481117037091603], (4808940926778034213,4810350864294758856], (-7508256920307222829,-7506372018227268626], (-7104590653728972577,-7104546570237712729], (3158009800098518496,3172651438220766183], (-2615481117037091603,-2611069890590917549], (-5878097817437987368,-5870054111151365631], (-2547658065527858190,-2544088936726049385], (232652608016417486,239394377432130766], (3154311195118940026,3158009800098518496]]] Validation failed in /192.168.10.63 (progress: 1%)
> [2018-05-25 19:31:50,027] Repair session 01b3f061-6050-11e8-a3b7-9d0793eab507 for range [(2424051311739332070,2424772330724108233], (6848066208555197,10521229928033262], (992385332284940308,1000066900542109637], (4418797036920007266,4421783585221695744], (-5417725796037372830,-5412149532100548404], (178766242164281045,191217736969025363], (-3802006636037441627,-3796416071827586080], (5683533739750457455,5688298632819249302], (3653327414143088744,3655860906328373441], (3655860906328373441,3657219071532471378], (5746716543928841040,5753897313199191356], (-7506372018227268626,-7477180353912675682], (1911795960615895165,1921474545637686707], (4421783585221695744,4445104759712094068], (-4428987737460108139,-4413904067417968038], (5680321325075541449,5683533739750457455]] failed with error [repair #01b3f061-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(2424051311739332070,2424772330724108233], (6848066208555197,10521229928033262], (992385332284940308,1000066900542109637], (4418797036920007266,4421783585221695744], (-5417725796037372830,-5412149532100548404], (178766242164281045,191217736969025363], (-3802006636037441627,-3796416071827586080], (5683533739750457455,5688298632819249302], (3653327414143088744,3655860906328373441], (3655860906328373441,3657219071532471378], (5746716543928841040,5753897313199191356], (-7506372018227268626,-7477180353912675682], (1911795960615895165,1921474545637686707], (4421783585221695744,4445104759712094068], (-4428987737460108139,-4413904067417968038], (5680321325075541449,5683533739750457455]]] Validation failed in /192.168.10.63 (progress: 1%)
> [2018-05-25 19:31:50,065] Repair session 01d226c2-6050-11e8-a3b7-9d0793eab507 for range [(731483217573828589,749016052425471844], (3349217091766639630,3355743728768043539], (8297509817744988677,8299811671851037140], (-1080064213437365415,-1067683134584617984], (-8988387420898594746,-8988256206650322851], (-1083473978088553649,-1080064213437365415], (-7068314886788869981,-7062826172876507507], (8299811671851037140,8306379796303668520], (-8500393685425499630,-8500127431270773970], (9077374236600850244,9080101637323836166], (9080101637323836166,9095536755598180114], (-2759657072078827823,-2750629632199441038], (-7938459356954944009,-7933123149264580832], (1759642905348136701,1772996641768793656], (-2788441126655538224,-2774970527117004032], (-7070810217579746608,-7068314886788869981], (-7959560447639828128,-7938459356954944009], (-7679921498492428955,-7664015662435807775]] failed with error [repair #01d226c2-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(731483217573828589,749016052425471844], (3349217091766639630,3355743728768043539], (8297509817744988677,8299811671851037140], (-1080064213437365415,-1067683134584617984], (-8988387420898594746,-8988256206650322851], (-1083473978088553649,-1080064213437365415], (-7068314886788869981,-7062826172876507507], (8299811671851037140,8306379796303668520], (-8500393685425499630,-8500127431270773970], (9077374236600850244,9080101637323836166], (9080101637323836166,9095536755598180114], (-2759657072078827823,-2750629632199441038], (-7938459356954944009,-7933123149264580832], (1759642905348136701,1772996641768793656], (-2788441126655538224,-2774970527117004032], (-7070810217579746608,-7068314886788869981], (-7959560447639828128,-7938459356954944009], (-7679921498492428955,-7664015662435807775]]] Validation failed in /192.168.8.63 (progress: 2%)
> [2018-05-25 19:32:24,797] Repair session 01aff8c0-6050-11e8-a3b7-9d0793eab507 for range [(119626359210683992,128454334208965433], (6169854579148936152,6189260921105966960], (8460580156771389602,8466680988634247357], (10521229928033262,11278848941988721], (6165215300562655515,6169854579148936152], (191217736969025363,212964375650430729], (-5297146550802223153,-5294434130239676253], (6189260921105966960,6193074220809370652], (-655425716305023073,-647730635946823030]] failed with error [repair #01aff8c0-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(119626359210683992,128454334208965433], (6169854579148936152,6189260921105966960], (8460580156771389602,8466680988634247357], (10521229928033262,11278848941988721], (6165215300562655515,6169854579148936152], (191217736969025363,212964375650430729], (-5297146550802223153,-5294434130239676253], (6189260921105966960,6193074220809370652], (-655425716305023073,-647730635946823030]]] Validation failed in /192.168.10.63 (progress: 2%)
> [2018-05-25 19:32:24,873] Repair session 0199d8b1-6050-11e8-a3b7-9d0793eab507 for range [(2708724319719658573,2710986923384204956], (6278718135193665575,6281813004301666161], (-8025315476660819134,-8015410683496661099], (2516704840921371424,2519633614752918103], (2519633614752918103,2526922953145276348], (8641102301927501454,8641256970223193109], (8643632109719583963,8645181823655307237], (-8015410683496661099,-8012293411840276426], (1368548173174048881,1373330457443776421], (5550121777767121,6848066208555197], (8641256970223193109,8643632109719583963], (-4201893423037098789,-4196287665648271477], (2692054381245703566,2708724319719658573], (-4208139091663389178,-4201893423037098789], (6281813004301666161,6282606461503930756], (-3470325001213070915,-3465759276556337455], (-4196287665648271477,-4185162268982289501], (-5006305410789315624,-5000646423000423501], (2714363942918413158,2722577239100121227], (5692402142504566885,5693342630493279303], (2710986923384204956,2714363942918413158], (5688298632819249302,5692402142504566885]] failed with error [repair #0199d8b1-6050-11e8-a3b7-9d0793eab507 on keyspace/table, [(2708724319719658573,2710986923384204956], (6278718135193665575,6281813004301666161], (-8025315476660819134,-8015410683496661099], (2516704840921371424,2519633614752918103], (2519633614752918103,2526922953145276348], (8641102301927501454,8641256970223193109], (8643632109719583963,8645181823655307237], (-8015410683496661099,-8012293411840276426], (1368548173174048881,1373330457443776421], (5550121777767121,6848066208555197], (8641256970223193109,8643632109719583963], (-4201893423037098789,-4196287665648271477], (2692054381245703566,2708724319719658573], (-4208139091663389178,-4201893423037098789], (6281813004301666161,6282606461503930756], (-3470325001213070915,-3465759276556337455], (-4196287665648271477,-4185162268982289501], (-5006305410789315624,-5000646423000423501], (2714363942918413158,2722577239100121227], (5692402142504566885,5693342630493279303], (2710986923384204956,2714363942918413158], (5688298632819249302,5692402142504566885]]] Validation failed in /192.168.8.65 (progress: 2%)
> Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException
> Cassandra has shutdown.
> error: [2018-05-25 19:36:47,652] JMX connection closed. You should check server log for repair status of keyspace keyspace(Subsequent keyspaces are
> not going to be repaired).
> -- StackTrace --
> May 25, 2018 7:36:47 PM ClientCommunicatorAdmin Checker-run
> WARNING: Failed to check connection: java.net.SocketException: Connection reset
> java.io.IOException: [2018-05-25 19:36:47,652] JMX connection closed. You should check server log for repair status of keyspace keyspace(Subsequent
> keyspaces are not going to be repaired).
> at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:98)
> at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
> at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
> at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
> at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
> at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
> at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
> at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
> at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
> at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:199)
> at java.lang.Thread.run(Thread.java:748)
> May 25, 2018 7:36:47 PM ClientCommunicatorAdmin Checker-run
> WARNING: stopping
> {code}
> Here is the log on one of the nodes where validation fails.
> {code:java}
> INFO [AntiEntropyStage:1] 2018-05-25 19:23:10,548 Validator.java:281 - [repair #01cf67a1-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree to /192.168.10.65 for pr$
> INFO [AntiEntropyStage:1] 2018-05-25 19:26:17,161 Validator.java:281 - [repair #01828020-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree to /192.168.10.65 for pr$
> INFO [AntiEntropyStage:1] 2018-05-25 19:26:23,909 Validator.java:281 - [repair #019dd051-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree to /192.168.10.65 for pr$
> INFO [AntiEntropyStage:1] 2018-05-25 19:28:15,118 Validator.java:281 - [repair #01c52e71-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree to /192.168.10.65 for pr$
> INFO [GossipTasks:1] 2018-05-25 19:30:23,087 Gossiper.java:1034 - InetAddress /192.168.10.65 is now DOWN
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:31,093 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:31,281 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [RequestResponseStage-4] 2018-05-25 19:30:31,320 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-3] 2018-05-25 19:30:31,320 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-2] 2018-05-25 19:30:31,320 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-1] 2018-05-25 19:30:31,320 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-5] 2018-05-25 19:30:31,320 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [AntiEntropyStage:1] 2018-05-25 19:30:49,172 Validator.java:281 - [repair #01860291-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree to /192.168.10.65 for pr$
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:49,188 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:54,188 OutboundTcpConnection.java:569 - Cannot handshake version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:54,188 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:59,188 OutboundTcpConnection.java:569 - Cannot handshake version with /192.168.10.65
> INFO [GossipTasks:1] 2018-05-25 19:31:03,247 Gossiper.java:1034 - InetAddress /192.168.10.65 is now DOWN
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:10,250 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:12,237 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [RequestResponseStage-7] 2018-05-25 19:31:12,712 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-9] 2018-05-25 19:31:12,712 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-13] 2018-05-25 19:31:12,712 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [GossipTasks:1] 2018-05-25 19:31:37,252 Gossiper.java:1034 - InetAddress /192.168.10.65 is now DOWN
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:45,254 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:48,759 OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,021 Validator.java:268 - Failed creating a merkle tree for [repair #01c26f52-6050-11e8-a3b7-9d0793eab507 on keyspace/$
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,022 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:7,1,main]java.lang.RuntimeException: Parent repair session with id = 0103da40-6050-11e8-a3b7-9d0793eab507 has failed. at org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(ActiveRepairService.java:412) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1459) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1366) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:955) ~[apache-cassandra-3.11.2.jar:3.11.2] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171
> ]INFO [RequestResponseStage-2] 2018-05-25 19:31:49,025 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-3] 2018-05-25 19:31:49,025 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> INFO [RequestResponseStage-1] 2018-05-25 19:31:49,039 Gossiper.java:1019 - InetAddress /192.168.10.65 is now UP
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,817 Validator.java:268 - Failed creating a merkle tree for [repair #01b3f061-6050-11e8-a3b7-9d0793eab507 on keyspace/$
> ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,817 CassandraDaemon.java:228 - Exception in thread Thread[ValidationExecutor:7,1,main]java.lang.RuntimeException: Parent repair session with id = 0103da40-6050-11e8-a3b7-9d0793eab507 has failed. at org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(ActiveRepairService.java:412) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1459) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1366) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:955) ~[apache-cassandra-3.11.2.jar:3.11.2] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
> {code}
> 192.168.10.65 is the node where I started the repair. It looks like this node goes down before the merkle tree creation failure occurs? The debug log on the repair node is full of the below and doesn't help me much.
> {code:java}
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,646 MerkleTree.java:295 - (17) Hashing sub-ranges [#<TreeRange (8732300281801533308,8732300339552037321] depth=18>, #<TreeRange (8732300339552037321,8732300397302541334] depth=18>] for #<TreeRange (8732300281801533308,8732300397302541334] depth=17> divided by midpoint 8732300339552037321
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:311 - (17) Inconsistent digest on left sub-range #<TreeRange (8732300281801533308,8732300339552037321] depth=18>: [#<Leaf [16cd9a47184232c7ed028a9d4546332d8a8c8ff83526f8f274997592eecc722d]>, #<Leaf [97b8f2fd61c1130ed2cfd8c2db52afdea59b2cabb5f39aed967cf8f2539f08b8]>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:333 - (17) Inconsistent digest on right sub-range #<TreeRange (8732300339552037321,8732300397302541334] depth=18>: [#<Leaf [fcf6daa5b5124a1e099e7776475aff22f2befedd88dc7b3e4277b92fd3115833]>, #<Leaf []>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (17) Fully inconsistent range [#<TreeRange (8732300281801533308,8732300339552037321] depth=18>, #<TreeRange (8732300339552037321,8732300397302541334] depth=18>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (16) Fully inconsistent range [#<TreeRange (8732300166300525283,8732300281801533308] depth=17>, #<TreeRange (8732300281801533308,8732300397302541334] depth=17>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (15) Fully inconsistent range [#<TreeRange (8732299935298509232,8732300166300525283] depth=16>, #<TreeRange (8732300166300525283,8732300397302541334] depth=16>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (14) Fully inconsistent range [#<TreeRange (8732299473294477131,8732299935298509232] depth=15>, #<TreeRange (8732299935298509232,8732300397302541334] depth=15>]
> DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (13) Fully inconsistent range [#<TreeRange (8732298549286412929,8732299473294477131] depth=14>, #<TreeRange (8732299473294477131,8732300397302541334] depth=14>]
> {code}
> Really at a loss of how to repair this table at this point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org