You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 曹志富 <ca...@gmail.com> on 2015/09/01 16:22:48 UTC
Re: abnormal log after remove a node
Just restart all of the c* node
--------------------------------------
Ranger Tsao
2015-08-25 18:17 GMT+08:00 Alain RODRIGUEZ <ar...@gmail.com>:
> Hi, I am facing the same issue on 2.0.16.
>
> Did you solve this ? How ?
>
> I plan to try a rolling restart and see if gossip state recover from this.
>
> C*heers,
>
> Alain
>
> 2015-06-19 11:40 GMT+02:00 曹志富 <ca...@gmail.com>:
>
>> I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node from
>> this cluster using nodetool decommission.
>>
>> But tody I find some log like this:
>>
>> INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 -
>> InetAddress /172.19.105.41 is now DOWN
>> INFO [GossipStage:1] 2015-06-19 17:38:05,617 StorageService.java:1885 -
>> Removing tokens [-1014432261309809702, -1055322450438958612,
>> -1120728727235087395, -1191392141261832305, -1203676771883970142,
>> -1215563040745505837, -1215648909329054362, -1269531760567530381,
>> -1278047879489577908, -1313427877031136549, -1342822572958042617,
>> -1350792764922315814, -1383390744017639599, -139000372807970456,
>> -140827955201469664, -1631551789771606023, -1633789813430312609,
>> -1795528665156349205, -1836619444785023397, -1879127294549041822,
>> -1962337787208890426, -2022309807234530256, -2033402140526360327,
>> -2089413865145942100, -210961549458416802, -2148530352195763113,
>> -2184481573787758786, -2222610790268720205, -2340762266634834427,
>> -2513416003567685694, -2520971378752190013, -2596695976621541808,
>> -2620636796023437199, -2640378596436678113, -2679143017361311011,
>> -2721176590519112233, -2749213392354746126, -279267896827516626,
>> -2872377759991294853, -2904711688111888325, -290489381926812623,
>> -3000574339499272616, -301428600802598523, -3019280155316984595,
>> -3024451041907074275, -3056898917375012425, -3161300347260716852,
>> -3166392383659271772, -3327634380871627036, -3530685865340274372,
>> -3563112657791369745, -366930313427781469, -3729582520450700795,
>> -3901838244986519991, -4065326606010524312, -4174346928341550117,
>> -4184239233207315432, -4204369933734181327, -4206479093137814808,
>> -421410317165821100, -4311166118017934135, -4407123461118340117,
>> -4466364858622123151, -4466939645485100087, -448955147512581975,
>> -4587780638857304626, -4649897584350376674, -4674234125365755024
>> , -4833801201210885896, -4857586579802212277, -4868896650650107463,
>> -4980063310159547694, -4983471821416248610, -4992846054037653676,
>> -5026994389965137674, -514302500353679181
>> 0, -5198414516309928594, -5245363745777287346, -5346838390293957674,
>> -5374413419545696184, -5427881744040857637, -5453876964430787287,
>> -5491923669475601173, -55219734138599212
>> 6, -5523011502670737422, -5537121117160410549, -5557015938925208697,
>> -5572489682738121748, -5745899409803353484, -5771239101488682535,
>> -5893479791287484099, -59766730414807540
>> 44, -6014643892406938367, -6086002438656595783, -6129360679394503700,
>> -6224240257573911174, -6290393495130499466, -6378712056928268929,
>> -6430306056990093461, -6800188263839065
>> 013, -6912720411187525051, -7160327814305587432, -7175004328733776324,
>> -7272070430660252577, -7307945744786025148, -742448651973108101,
>> -7539255117639002578, -7657460716997978
>> 94, -7846698077070579798, -7870621904906244395, -7900841391761900719,
>> -7918145426423910061, -7936795453892692473, -8070255024778921411,
>> -8086888710627677669, -8124855925323654
>> 631, -8175270408138820500, -8271197636596881168, -8336685710406477123,
>> -8466220397076441627, -8534337908154758270, -8550484400487603561,
>> -862246738021989870, -8727219287242892
>> 185, -8895705475282612927, -8921801772904834063, -9057266752652143883,
>> -9059183540698454288, -9067986437682229598, -9148183367896132028,
>> -962208188860606543, 10859447725819218
>> 30, 1189775396643491793, 1253728955879686947, 1389982523380382228,
>> 1429632314664544045, 143610053770130548, 150118120072602242,
>> 1575692041584712198, 1624575905722628764, 17894
>> 76212785155173, 1995296121962835019, 2041217364870030239,
>> 2120277336231792146, 2124445736743406711, 2154979704292433983,
>> 2340726755918680765, 23481654796845972, 23620268084352
>> 24407, 2366144489007464626, 2381492708106933027, 2398868971489617398,
>> 2427315953339163528, 2433999003913998534, 2633074510238705620,
>> 266659839023809792, 2677817641360639089, 2
>> 719725410894526151, 2751925111749406683, 2815703589803785617,
>> 3041515796379693113, 3044903149214270978, 3094954503756703989,
>> 3243933267690865263, 3246086646486800371, 33270068
>> 97333869434, 3393657685587750192, 3395065499228709345,
>> 3426126123948029459, 3500469615600510698, 3644011364716880512,
>> 3693249207133187620, 3776164494954636918, 385555780676797
>> 8035, 3872151295451662867, 3937077827707223414, 4041082935346014761,
>> 4060208918173638435, 4086747843759164940, 4165638694482690057,
>> 4203996339238989224, 4220155275330961826, 4
>> 366784953339236686, 4390116924352514616, 4391225331964772681,
>> 4392419346255765958, 4448400054980766409, 4463335839328115373,
>> 4547306976104362915, 4588174843388248100, 48438580
>> 67983993745, 4912719175808770608, 499628843707992459,
>> 5004392861473086088, 5021047773702107258, 510226752691159107,
>> 5109551630357971118, 5157669927051121583, 51627694176199618
>> 24, 5238710860488961530, 5245958115092331518, 5302459768185143407,
>> 5373077323749320571, 5445982956737768774, 5526076427753104565,
>> 5531878975169972758, 5590672474842108747, 561
>> 8238086143944892, 5645763748154253201, 5648082473497629258,
>> 5799608283794045232, 5968931466409317704, 6080339666926312644,
>> 6222992739052178144, 6329332485451402638, 6372655016
>> 269952381, 6450034006536146156, 6526155371478962031, 6548600217631455803,
>> 6591556769219166621, 6658093966781219462, 6673583970197595202,
>> 6721108087363914888, 67250010230161815
>> 66, 6755422648396704654, 6771784849502850301, 6894457688282530581,
>> 6935906357620278103, 7290035612348735845, 7391177400231418753,
>> 7392159536582094629, 7416025635188268574, 777
>> 2839976879697009, 7779944872726958967, 7813009726137267594,
>> 7828693936020907346, 783798051009400395, 794321710728449974,
>> 7969419520390622697, 8032162688041296427, 813984063616
>> 8254157, 8246865194299129430, 8318906537193659373, 842843196067279262,
>> 8540875087395336317, 8666350659048928104, 8697352861510764965,
>> 8801847806932416388, 8828239026404690350,
>> 8874223097578484531, 8880048920270623806, 8882247680076975559,
>> 89230101686533845, 8977307252904558369, 9037550178585102557,
>> 9199020538928645866] for /172.19.105.41
>>
>>
>> ant this log info a few minutes one time
>>
>>
>> --------------------------------------
>> Ranger Tsao
>>
>
>
Re: abnormal log after remove a node
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi,
I finally did the exact same thing before receiving the answer.
I guess this will remain FTR :).
Thanks though !
Alain
2015-09-01 16:22 GMT+02:00 曹志富 <ca...@gmail.com>:
> Just restart all of the c* node
>
> --------------------------------------
> Ranger Tsao
>
> 2015-08-25 18:17 GMT+08:00 Alain RODRIGUEZ <ar...@gmail.com>:
>
>> Hi, I am facing the same issue on 2.0.16.
>>
>> Did you solve this ? How ?
>>
>> I plan to try a rolling restart and see if gossip state recover from this.
>>
>> C*heers,
>>
>> Alain
>>
>> 2015-06-19 11:40 GMT+02:00 曹志富 <ca...@gmail.com>:
>>
>>> I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node
>>> from this cluster using nodetool decommission.
>>>
>>> But tody I find some log like this:
>>>
>>> INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 -
>>> InetAddress /172.19.105.41 is now DOWN
>>> INFO [GossipStage:1] 2015-06-19 17:38:05,617 StorageService.java:1885 -
>>> Removing tokens [-1014432261309809702, -1055322450438958612,
>>> -1120728727235087395, -1191392141261832305, -1203676771883970142,
>>> -1215563040745505837, -1215648909329054362, -1269531760567530381,
>>> -1278047879489577908, -1313427877031136549, -1342822572958042617,
>>> -1350792764922315814, -1383390744017639599, -139000372807970456,
>>> -140827955201469664, -1631551789771606023, -1633789813430312609,
>>> -1795528665156349205, -1836619444785023397, -1879127294549041822,
>>> -1962337787208890426, -2022309807234530256, -2033402140526360327,
>>> -2089413865145942100, -210961549458416802, -2148530352195763113,
>>> -2184481573787758786, -2222610790268720205, -2340762266634834427,
>>> -2513416003567685694, -2520971378752190013, -2596695976621541808,
>>> -2620636796023437199, -2640378596436678113, -2679143017361311011,
>>> -2721176590519112233, -2749213392354746126, -279267896827516626,
>>> -2872377759991294853, -2904711688111888325, -290489381926812623,
>>> -3000574339499272616, -301428600802598523, -3019280155316984595,
>>> -3024451041907074275, -3056898917375012425, -3161300347260716852,
>>> -3166392383659271772, -3327634380871627036, -3530685865340274372,
>>> -3563112657791369745, -366930313427781469, -3729582520450700795,
>>> -3901838244986519991, -4065326606010524312, -4174346928341550117,
>>> -4184239233207315432, -4204369933734181327, -4206479093137814808,
>>> -421410317165821100, -4311166118017934135, -4407123461118340117,
>>> -4466364858622123151, -4466939645485100087, -448955147512581975,
>>> -4587780638857304626, -4649897584350376674, -4674234125365755024
>>> , -4833801201210885896, -4857586579802212277, -4868896650650107463,
>>> -4980063310159547694, -4983471821416248610, -4992846054037653676,
>>> -5026994389965137674, -514302500353679181
>>> 0, -5198414516309928594, -5245363745777287346, -5346838390293957674,
>>> -5374413419545696184, -5427881744040857637, -5453876964430787287,
>>> -5491923669475601173, -55219734138599212
>>> 6, -5523011502670737422, -5537121117160410549, -5557015938925208697,
>>> -5572489682738121748, -5745899409803353484, -5771239101488682535,
>>> -5893479791287484099, -59766730414807540
>>> 44, -6014643892406938367, -6086002438656595783, -6129360679394503700,
>>> -6224240257573911174, -6290393495130499466, -6378712056928268929,
>>> -6430306056990093461, -6800188263839065
>>> 013, -6912720411187525051, -7160327814305587432, -7175004328733776324,
>>> -7272070430660252577, -7307945744786025148, -742448651973108101,
>>> -7539255117639002578, -7657460716997978
>>> 94, -7846698077070579798, -7870621904906244395, -7900841391761900719,
>>> -7918145426423910061, -7936795453892692473, -8070255024778921411,
>>> -8086888710627677669, -8124855925323654
>>> 631, -8175270408138820500, -8271197636596881168, -8336685710406477123,
>>> -8466220397076441627, -8534337908154758270, -8550484400487603561,
>>> -862246738021989870, -8727219287242892
>>> 185, -8895705475282612927, -8921801772904834063, -9057266752652143883,
>>> -9059183540698454288, -9067986437682229598, -9148183367896132028,
>>> -962208188860606543, 10859447725819218
>>> 30, 1189775396643491793, 1253728955879686947, 1389982523380382228,
>>> 1429632314664544045, 143610053770130548, 150118120072602242,
>>> 1575692041584712198, 1624575905722628764, 17894
>>> 76212785155173, 1995296121962835019, 2041217364870030239,
>>> 2120277336231792146, 2124445736743406711, 2154979704292433983,
>>> 2340726755918680765, 23481654796845972, 23620268084352
>>> 24407, 2366144489007464626, 2381492708106933027, 2398868971489617398,
>>> 2427315953339163528, 2433999003913998534, 2633074510238705620,
>>> 266659839023809792, 2677817641360639089, 2
>>> 719725410894526151, 2751925111749406683, 2815703589803785617,
>>> 3041515796379693113, 3044903149214270978, 3094954503756703989,
>>> 3243933267690865263, 3246086646486800371, 33270068
>>> 97333869434, 3393657685587750192, 3395065499228709345,
>>> 3426126123948029459, 3500469615600510698, 3644011364716880512,
>>> 3693249207133187620, 3776164494954636918, 385555780676797
>>> 8035, 3872151295451662867, 3937077827707223414, 4041082935346014761,
>>> 4060208918173638435, 4086747843759164940, 4165638694482690057,
>>> 4203996339238989224, 4220155275330961826, 4
>>> 366784953339236686, 4390116924352514616, 4391225331964772681,
>>> 4392419346255765958, 4448400054980766409, 4463335839328115373,
>>> 4547306976104362915, 4588174843388248100, 48438580
>>> 67983993745, 4912719175808770608, 499628843707992459,
>>> 5004392861473086088, 5021047773702107258, 510226752691159107,
>>> 5109551630357971118, 5157669927051121583, 51627694176199618
>>> 24, 5238710860488961530, 5245958115092331518, 5302459768185143407,
>>> 5373077323749320571, 5445982956737768774, 5526076427753104565,
>>> 5531878975169972758, 5590672474842108747, 561
>>> 8238086143944892, 5645763748154253201, 5648082473497629258,
>>> 5799608283794045232, 5968931466409317704, 6080339666926312644,
>>> 6222992739052178144, 6329332485451402638, 6372655016
>>> 269952381, 6450034006536146156, 6526155371478962031,
>>> 6548600217631455803, 6591556769219166621, 6658093966781219462,
>>> 6673583970197595202, 6721108087363914888, 67250010230161815
>>> 66, 6755422648396704654, 6771784849502850301, 6894457688282530581,
>>> 6935906357620278103, 7290035612348735845, 7391177400231418753,
>>> 7392159536582094629, 7416025635188268574, 777
>>> 2839976879697009, 7779944872726958967, 7813009726137267594,
>>> 7828693936020907346, 783798051009400395, 794321710728449974,
>>> 7969419520390622697, 8032162688041296427, 813984063616
>>> 8254157, 8246865194299129430, 8318906537193659373, 842843196067279262,
>>> 8540875087395336317, 8666350659048928104, 8697352861510764965,
>>> 8801847806932416388, 8828239026404690350,
>>> 8874223097578484531, 8880048920270623806, 8882247680076975559,
>>> 89230101686533845, 8977307252904558369, 9037550178585102557,
>>> 9199020538928645866] for /172.19.105.41
>>>
>>>
>>> ant this log info a few minutes one time
>>>
>>>
>>> --------------------------------------
>>> Ranger Tsao
>>>
>>
>>
>