How to get rid of "Found unknown server" for a decomissioned machine

[Adam Sjøgren <as...@koldfront.dk.INVALID>]

  Hi,


I am currently moving an HBase cluster onto new hardware, one machine
at a time.

So far I have moved over 11 of 25 servers.

After decomissioning the latest server, "hbase hbck" complains:

    ERROR: Found unknown server, some of the regions held by this server may not get assigned. Use HBCK2 scheduleRecoveries node19.example.com,16020,1711374038363 to recover.

I tried doing as instructed - HBCK2 outputs a pid and the web
interface shows the ServerCrashProcedure completing quickly.

However when I run "hbase hbck" again I get the same error and
suggestion.

I tried browsing HDFS to see if I could find any files with the name
of the machine in it, and I tried looking in ZooKeeper - no luck;
maybe I didn't look thoroughly enough.

Any suggestions on how to fix this?

I also tried running HBCK2 with "recoverUnknown", which also showed up
as completed in the web interface, but "hbase hbck" still outputs the
error.


  Best regards,

    Adam

-- 
 "Our hero has very high insurance premiums."               Adam Sjøgren
                                                       asjo@koldfront.dk

Re: How to get rid of "Found unknown server" for a decomissioned machine

[Adam Sjøgren <as...@koldfront.dk.INVALID>]

Adam writes:

>     ERROR: Found unknown server, some of the regions held by this server may not get assigned. Use HBCK2 scheduleRecoveries node19.example.com,16020,1711374038363 to recover.

A good colleague of mine suggested figuring out what table was
affected by running "scan 'hbase:meta'" in hbase shell.

That identified the unknown server being connected to a temporary
table, that I could delete, which solved the problem.

If it had been a table I couldn't just delete, I still don't know how
to solve the problem.


  Best regards,

    Adam

-- 
 "I carry it over my shoulder like a lance                  Adam Sjøgren
  A pole with a banner that says t-to-tolerance"       asjo@koldfront.dk