You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Kenworthy <an...@stackable.tech.INVALID> on 2025/04/24 13:41:56 UTC

HBase on Kubernetes: resolvable UI links

Hi,

I am running HBase clusters on Kubernetes and have been looking at how to
patch the code (version 2.6.1) so that the links on the master and
regionserver UIs are resolvable. My approach involves extending the
ServerName struct to include two new fields (e.g. externalHostname and
externalPort), which are filled with e.g. the node IP and externally
resolvable port, prepared by my operator and stored in custom config
settings. This means that I can use this information whenever a ServerName
object is retrieved for the active master, and I have patched the relevant
jsp pages to use this information instead of the internal name. However, it
is a) fairly invasive, and b) not a complete solution, as at some points in
the code (e.g. the regionServers collection in RegionServerTracker) only
the hostname of the struct is used and not the whole struct (so additional
fields are not available). I'm aware that things will change as of 4.0.0
anyway, when some of the meta data will be stored in an HBase table instead
of ZooKeeper. Has anyone else started thinking about or working on this? Is
there a work-in-progress that we could support upstream?

Andrew

Re: HBase on Kubernetes: resolvable UI links

Posted by Andrew Kenworthy <an...@stackable.tech.INVALID>.
Hi Nick,

Many thanks for your response. My first attempt was to use
hbase.master.{hostname|port}, hbase.unsafe.regionserver.hostname and
hbase.regionserver.port to set the externally reachable endpoints, but ran
into this
<https://github.com/apache/hbase/blob/branch-2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L766>
issue (we have turned off reverse DNS lookups to ensure the names of
RegionServers are resolved to their hostnames instead of IP addresses), and
using node IPs does not work in Kubernetes for RPC calls if the pods do not
have access to the host network. I tried to separate those two things
(client accessibility and RPC calls) but without success: maybe I will
revisit that and see if I can get it working.

Regards,
Andrew

On Thu, Apr 24, 2025 at 4:11 PM Nick Dimiduk <nd...@apache.org> wrote:

> Hi Andrew,
>
> I believe that we handle this issue by wrapping the UIs in a rewriting
> proxy. We have to do the same for the DataNodes. It would be nice to
> have native support for this kind of thing, but I'm not familiar with
> whatever guidance the Kubernetes community provides. Your approach
> does seem like the logical choice, but I also agree it feels
> heavy-handed. Hopefully someone else can chime in with their
> experiences.
>
> Your proposal reminds me vaguely of the config
> `hbase.unsafe.regionserver.hostname` and HBASE-12954. As I understand
> it, that mechanism monkeys with the hostname used in the RPC layer.
> Maybe it makes sense to also use it for the web UIs? Or maybe not,
> because routing internal RPC requests through the external-facing
> network stack is wasteful.
>
> Thanks,
> Nick
>
> On Thu, Apr 24, 2025 at 3:42 PM Andrew Kenworthy
> <an...@stackable.tech.invalid> wrote:
> >
> > Hi,
> >
> > I am running HBase clusters on Kubernetes and have been looking at how to
> > patch the code (version 2.6.1) so that the links on the master and
> > regionserver UIs are resolvable. My approach involves extending the
> > ServerName struct to include two new fields (e.g. externalHostname and
> > externalPort), which are filled with e.g. the node IP and externally
> > resolvable port, prepared by my operator and stored in custom config
> > settings. This means that I can use this information whenever a
> ServerName
> > object is retrieved for the active master, and I have patched the
> relevant
> > jsp pages to use this information instead of the internal name. However,
> it
> > is a) fairly invasive, and b) not a complete solution, as at some points
> in
> > the code (e.g. the regionServers collection in RegionServerTracker) only
> > the hostname of the struct is used and not the whole struct (so
> additional
> > fields are not available). I'm aware that things will change as of 4.0.0
> > anyway, when some of the meta data will be stored in an HBase table
> instead
> > of ZooKeeper. Has anyone else started thinking about or working on this?
> Is
> > there a work-in-progress that we could support upstream?
> >
> > Andrew
>

Re: HBase on Kubernetes: resolvable UI links

Posted by Nick Dimiduk <nd...@apache.org>.
Hi Andrew,

I believe that we handle this issue by wrapping the UIs in a rewriting
proxy. We have to do the same for the DataNodes. It would be nice to
have native support for this kind of thing, but I'm not familiar with
whatever guidance the Kubernetes community provides. Your approach
does seem like the logical choice, but I also agree it feels
heavy-handed. Hopefully someone else can chime in with their
experiences.

Your proposal reminds me vaguely of the config
`hbase.unsafe.regionserver.hostname` and HBASE-12954. As I understand
it, that mechanism monkeys with the hostname used in the RPC layer.
Maybe it makes sense to also use it for the web UIs? Or maybe not,
because routing internal RPC requests through the external-facing
network stack is wasteful.

Thanks,
Nick

On Thu, Apr 24, 2025 at 3:42 PM Andrew Kenworthy
<an...@stackable.tech.invalid> wrote:
>
> Hi,
>
> I am running HBase clusters on Kubernetes and have been looking at how to
> patch the code (version 2.6.1) so that the links on the master and
> regionserver UIs are resolvable. My approach involves extending the
> ServerName struct to include two new fields (e.g. externalHostname and
> externalPort), which are filled with e.g. the node IP and externally
> resolvable port, prepared by my operator and stored in custom config
> settings. This means that I can use this information whenever a ServerName
> object is retrieved for the active master, and I have patched the relevant
> jsp pages to use this information instead of the internal name. However, it
> is a) fairly invasive, and b) not a complete solution, as at some points in
> the code (e.g. the regionServers collection in RegionServerTracker) only
> the hostname of the struct is used and not the whole struct (so additional
> fields are not available). I'm aware that things will change as of 4.0.0
> anyway, when some of the meta data will be stored in an HBase table instead
> of ZooKeeper. Has anyone else started thinking about or working on this? Is
> there a work-in-progress that we could support upstream?
>
> Andrew