You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Alexandre Fouché <af...@onfocus.io> on 2017/03/09 14:34:00 UTC
How to reuse tablet server UUID, or removing old one
Hi all
I have searched and searched, but could not find how to tell that a
previous tablet server uuid is to be removed
I had to replace disks on tabletserver5, so i deleted all data, since there
were replicas on other servers. Now that i restarted Kudu on tabletserver5
with empty data, it initialised fine, but since (i saw afterwards on webUI)
it has a new UUID, it is recognised as a new server, and it does not resync
its tablets replcas with other tablet servers (edited). It has the same
hostname as before but different UUID
And in WebUI, i see the same tabletserver5 with the previous UUID marked as
‘dead'
How can i tel Kudu to completely remove the dead tabletserver5 UUID and
populate the new tabletserver5 UUID instead ?
the `kudu` command line tool does not seem to allow to delete a tablet
server UUID, or decommission
so how ?
Or other way, how can i recreate an empty Kudu tablet server reusing my old
UUID ?
Re: How to reuse tablet server UUID, or removing old one
Posted by Alexandre Fouché <af...@onfocus.io>.
Greaat, Thank you all very much !
2017-03-10 1:59 GMT+01:00 Todd Lipcon <to...@cloudera.com>:
> It should disappear when you next restart the masters. They don't persist
> the list of tservers, but rather learn about them dynamically when they
> come up.
>
> -Todd
>
> On Thu, Mar 9, 2017 at 11:05 AM, Alexandre Fouché <af...@onfocus.io>
> wrote:
>
>> Oh ok, so since i need to do replace partitionning on all other nodes, i
>> suppose tabletserver5 will get populated when i delete other tabletservers
>> one by one.
>>
>> And indeed, now i see from the webUI, that all my tablets still have 3
>> replicas, so Kudu must have ensured a replication of 3 when tabletserver5
>> was considered dead. (i wonder how much time after though)
>>
>> Yet, is it possible to have the masters forget about the dead tablet
>> server UUID, so that it does not show up anymore in the webUI ? Or will it
>> disapperar after a while or a Kudu restart maybe ?
>>
>>
>> 2017-03-09 17:48 GMT+01:00 Jean-Daniel Cryans <jd...@apache.org>:
>>
>>> Hi Alexandre,
>>>
>>> Tablet replicas are not tied to a UUID, so removing or reusing one
>>> wouldn't achieve what you want. The main thing missing here is that Kudu
>>> doesn't do tablet re-balancing at runtime, so tabletserver5 will get
>>> tablets the next time a node dies or if you create new tables.
>>>
>>> Obviously that's something we'd like to address but nobody has come
>>> around to doing it so far.
>>>
>>> Sincèrement,
>>>
>>> J-D
>>>
>>> On Thu, Mar 9, 2017 at 6:34 AM, Alexandre Fouché <af...@onfocus.io>
>>> wrote:
>>>
>>>> Hi all
>>>>
>>>> I have searched and searched, but could not find how to tell that a
>>>> previous tablet server uuid is to be removed
>>>> I had to replace disks on tabletserver5, so i deleted all data, since
>>>> there were replicas on other servers. Now that i restarted Kudu on
>>>> tabletserver5 with empty data, it initialised fine, but since (i saw
>>>> afterwards on webUI) it has a new UUID, it is recognised as a new server,
>>>> and it does not resync its tablets replcas with other tablet servers
>>>> (edited). It has the same hostname as before but different UUID
>>>>
>>>> And in WebUI, i see the same tabletserver5 with the previous UUID
>>>> marked as ‘dead'
>>>>
>>>> How can i tel Kudu to completely remove the dead tabletserver5 UUID and
>>>> populate the new tabletserver5 UUID instead ?
>>>>
>>>> the `kudu` command line tool does not seem to allow to delete a tablet
>>>> server UUID, or decommission
>>>> so how ?
>>>>
>>>> Or other way, how can i recreate an empty Kudu tablet server reusing my
>>>> old UUID ?
>>>>
>>>
>>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: How to reuse tablet server UUID, or removing old one
Posted by Todd Lipcon <to...@cloudera.com>.
It should disappear when you next restart the masters. They don't persist
the list of tservers, but rather learn about them dynamically when they
come up.
-Todd
On Thu, Mar 9, 2017 at 11:05 AM, Alexandre Fouché <af...@onfocus.io>
wrote:
> Oh ok, so since i need to do replace partitionning on all other nodes, i
> suppose tabletserver5 will get populated when i delete other tabletservers
> one by one.
>
> And indeed, now i see from the webUI, that all my tablets still have 3
> replicas, so Kudu must have ensured a replication of 3 when tabletserver5
> was considered dead. (i wonder how much time after though)
>
> Yet, is it possible to have the masters forget about the dead tablet
> server UUID, so that it does not show up anymore in the webUI ? Or will it
> disapperar after a while or a Kudu restart maybe ?
>
>
> 2017-03-09 17:48 GMT+01:00 Jean-Daniel Cryans <jd...@apache.org>:
>
>> Hi Alexandre,
>>
>> Tablet replicas are not tied to a UUID, so removing or reusing one
>> wouldn't achieve what you want. The main thing missing here is that Kudu
>> doesn't do tablet re-balancing at runtime, so tabletserver5 will get
>> tablets the next time a node dies or if you create new tables.
>>
>> Obviously that's something we'd like to address but nobody has come
>> around to doing it so far.
>>
>> Sincèrement,
>>
>> J-D
>>
>> On Thu, Mar 9, 2017 at 6:34 AM, Alexandre Fouché <af...@onfocus.io>
>> wrote:
>>
>>> Hi all
>>>
>>> I have searched and searched, but could not find how to tell that a
>>> previous tablet server uuid is to be removed
>>> I had to replace disks on tabletserver5, so i deleted all data, since
>>> there were replicas on other servers. Now that i restarted Kudu on
>>> tabletserver5 with empty data, it initialised fine, but since (i saw
>>> afterwards on webUI) it has a new UUID, it is recognised as a new server,
>>> and it does not resync its tablets replcas with other tablet servers
>>> (edited). It has the same hostname as before but different UUID
>>>
>>> And in WebUI, i see the same tabletserver5 with the previous UUID marked
>>> as ‘dead'
>>>
>>> How can i tel Kudu to completely remove the dead tabletserver5 UUID and
>>> populate the new tabletserver5 UUID instead ?
>>>
>>> the `kudu` command line tool does not seem to allow to delete a tablet
>>> server UUID, or decommission
>>> so how ?
>>>
>>> Or other way, how can i recreate an empty Kudu tablet server reusing my
>>> old UUID ?
>>>
>>
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: How to reuse tablet server UUID, or removing old one
Posted by Alexandre Fouché <af...@onfocus.io>.
Oh ok, so since i need to do replace partitionning on all other nodes, i
suppose tabletserver5 will get populated when i delete other tabletservers
one by one.
And indeed, now i see from the webUI, that all my tablets still have 3
replicas, so Kudu must have ensured a replication of 3 when tabletserver5
was considered dead. (i wonder how much time after though)
Yet, is it possible to have the masters forget about the dead tablet server
UUID, so that it does not show up anymore in the webUI ? Or will it
disapperar after a while or a Kudu restart maybe ?
2017-03-09 17:48 GMT+01:00 Jean-Daniel Cryans <jd...@apache.org>:
> Hi Alexandre,
>
> Tablet replicas are not tied to a UUID, so removing or reusing one
> wouldn't achieve what you want. The main thing missing here is that Kudu
> doesn't do tablet re-balancing at runtime, so tabletserver5 will get
> tablets the next time a node dies or if you create new tables.
>
> Obviously that's something we'd like to address but nobody has come around
> to doing it so far.
>
> Sincèrement,
>
> J-D
>
> On Thu, Mar 9, 2017 at 6:34 AM, Alexandre Fouché <af...@onfocus.io>
> wrote:
>
>> Hi all
>>
>> I have searched and searched, but could not find how to tell that a
>> previous tablet server uuid is to be removed
>> I had to replace disks on tabletserver5, so i deleted all data, since
>> there were replicas on other servers. Now that i restarted Kudu on
>> tabletserver5 with empty data, it initialised fine, but since (i saw
>> afterwards on webUI) it has a new UUID, it is recognised as a new server,
>> and it does not resync its tablets replcas with other tablet servers
>> (edited). It has the same hostname as before but different UUID
>>
>> And in WebUI, i see the same tabletserver5 with the previous UUID marked
>> as ‘dead'
>>
>> How can i tel Kudu to completely remove the dead tabletserver5 UUID and
>> populate the new tabletserver5 UUID instead ?
>>
>> the `kudu` command line tool does not seem to allow to delete a tablet
>> server UUID, or decommission
>> so how ?
>>
>> Or other way, how can i recreate an empty Kudu tablet server reusing my
>> old UUID ?
>>
>
>
Re: How to reuse tablet server UUID, or removing old one
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Hi Alexandre,
Tablet replicas are not tied to a UUID, so removing or reusing one wouldn't
achieve what you want. The main thing missing here is that Kudu doesn't do
tablet re-balancing at runtime, so tabletserver5 will get tablets the next
time a node dies or if you create new tables.
Obviously that's something we'd like to address but nobody has come around
to doing it so far.
Sincèrement,
J-D
On Thu, Mar 9, 2017 at 6:34 AM, Alexandre Fouché <af...@onfocus.io> wrote:
> Hi all
>
> I have searched and searched, but could not find how to tell that a
> previous tablet server uuid is to be removed
> I had to replace disks on tabletserver5, so i deleted all data, since
> there were replicas on other servers. Now that i restarted Kudu on
> tabletserver5 with empty data, it initialised fine, but since (i saw
> afterwards on webUI) it has a new UUID, it is recognised as a new server,
> and it does not resync its tablets replcas with other tablet servers
> (edited). It has the same hostname as before but different UUID
>
> And in WebUI, i see the same tabletserver5 with the previous UUID marked
> as ‘dead'
>
> How can i tel Kudu to completely remove the dead tabletserver5 UUID and
> populate the new tabletserver5 UUID instead ?
>
> the `kudu` command line tool does not seem to allow to delete a tablet
> server UUID, or decommission
> so how ?
>
> Or other way, how can i recreate an empty Kudu tablet server reusing my
> old UUID ?
>