Leader address when behind external DNS

[Thomas Cooper <to...@gmail.com>]

Hi,

I am working on a prototype Kubernetes (K8s) operator for Storm. I have
everything up and running inside K8s, which in my case is an instance of
minikube [1] running on my local laptop. I have set up a service to point
to the Nimbus node (there is only 1 at the moment) and in my case I have
forwarded this service to a port on my dev machine (eg: localhost:6627),
but this could equally be mydomain.com:6627.

The issue of course is that Nimbus' internal address does not match this
external one. So when I run the storm client CLI on my local machine, the
client code [2] asks for the Nimbus leader info and returns the
internal address, which is what I want to happen for the workers etc but
not for this external client which needs to use localhost:6627.

I was wondering if there is already a way to deal with this situation? My
search through the docs and code hasn't found an obvious one.

I did think that one possible solution would be to provide some kind of
override on the external client side that would provide the nimbus.seeds
external equivalents and have the client use this new
"advertised.nimbus.seeds" config instead. They could be a seperate list in
the same order as the nimbus seeds and the external client could swap them
out like for like?

A more headyweight option may be to implement code in the Nimbus HA cluster
to forward certain admin commands to the leader (as every member of the
cluster should know who that is). If that functionality existed we
could then create a load balanced service that would fire the admin
commands at any Nimbus node and not need to return the Leader address to
the client at all.

I would love to hear people's opinions on the above (hopefully there is
already a solution to this).

[1] https://minikube.sigs.k8s.io/docs/
[2]
https://github.com/apache/storm/blob/0859cd191715a09f5c84e781e3d0e3ca89e60654/storm-client/src/jvm/org/apache/storm/utils/NimbusClient.java#L222

Cheers,

Tom Cooper
tomcooper.dev | @tomncooper <https://twitter.com/tomncooper>

Re: Leader address when behind external DNS

[Thomas Cooper <to...@gmail.com>]

Hi Rui,

In your case was you nimbus.seed config for the nimbus in the container
["localhost"]? Because that would work in your situation. The nimbus node
would return leader information to the external client with "localhost" as
the host name, which would match the "external" address for the command
line client.

I would at the very least like to get topology listing and jar
submission commands working, but they both share the same root issue, that
the leader info returned to the external client contains the
internal nimbus.seeds which may not match the external addresses.

I think I will try putting together a PoC with an override option for the
nimbus address and see how difficult that turns out to be.

Cheers,

Tom Cooper
tomcooper.dev | @tomncooper <https://twitter.com/tomncooper>


On Wed, 17 Mar 2021 at 15:08, Rui Abreu <ru...@gmail.com> wrote:

> Hi Tom,
>
> I recall using a docker container on my machine that was running Storm
> nimbus and Storm supervisors and through a port mapping between the
> container and my machine, I was able to upload topologies to the cluster
> through localhost:6627
> What kind of client operations are you referring to?
>
> On Sun, 14 Mar 2021 at 10:23, Thomas Cooper <to...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am working on a prototype Kubernetes (K8s) operator for Storm. I have
>> everything up and running inside K8s, which in my case is an instance of
>> minikube [1] running on my local laptop. I have set up a service to point
>> to the Nimbus node (there is only 1 at the moment) and in my case I have
>> forwarded this service to a port on my dev machine (eg: localhost:6627),
>> but this could equally be mydomain.com:6627.
>>
>> The issue of course is that Nimbus' internal address does not match this
>> external one. So when I run the storm client CLI on my local machine, the
>> client code [2] asks for the Nimbus leader info and returns the
>> internal address, which is what I want to happen for the workers etc but
>> not for this external client which needs to use localhost:6627.
>>
>> I was wondering if there is already a way to deal with this situation? My
>> search through the docs and code hasn't found an obvious one.
>>
>> I did think that one possible solution would be to provide some kind of
>> override on the external client side that would provide the nimbus.seeds
>> external equivalents and have the client use this new
>> "advertised.nimbus.seeds" config instead. They could be a seperate list in
>> the same order as the nimbus seeds and the external client could swap them
>> out like for like?
>>
>> A more headyweight option may be to implement code in the Nimbus HA
>> cluster to forward certain admin commands to the leader (as every member of
>> the cluster should know who that is). If that functionality existed we
>> could then create a load balanced service that would fire the admin
>> commands at any Nimbus node and not need to return the Leader address to
>> the client at all.
>>
>> I would love to hear people's opinions on the above (hopefully there is
>> already a solution to this).
>>
>> [1] https://minikube.sigs.k8s.io/docs/
>> [2]
>> https://github.com/apache/storm/blob/0859cd191715a09f5c84e781e3d0e3ca89e60654/storm-client/src/jvm/org/apache/storm/utils/NimbusClient.java#L222
>>
>> Cheers,
>>
>> Tom Cooper
>> tomcooper.dev | @tomncooper <https://twitter.com/tomncooper>
>>
>

Re: Leader address when behind external DNS

[Rui Abreu <ru...@gmail.com>]

Hi Tom,

I recall using a docker container on my machine that was running Storm
nimbus and Storm supervisors and through a port mapping between the
container and my machine, I was able to upload topologies to the cluster
through localhost:6627
What kind of client operations are you referring to?

On Sun, 14 Mar 2021 at 10:23, Thomas Cooper <to...@gmail.com> wrote:

> Hi,
>
> I am working on a prototype Kubernetes (K8s) operator for Storm. I have
> everything up and running inside K8s, which in my case is an instance of
> minikube [1] running on my local laptop. I have set up a service to point
> to the Nimbus node (there is only 1 at the moment) and in my case I have
> forwarded this service to a port on my dev machine (eg: localhost:6627),
> but this could equally be mydomain.com:6627.
>
> The issue of course is that Nimbus' internal address does not match this
> external one. So when I run the storm client CLI on my local machine, the
> client code [2] asks for the Nimbus leader info and returns the
> internal address, which is what I want to happen for the workers etc but
> not for this external client which needs to use localhost:6627.
>
> I was wondering if there is already a way to deal with this situation? My
> search through the docs and code hasn't found an obvious one.
>
> I did think that one possible solution would be to provide some kind of
> override on the external client side that would provide the nimbus.seeds
> external equivalents and have the client use this new
> "advertised.nimbus.seeds" config instead. They could be a seperate list in
> the same order as the nimbus seeds and the external client could swap them
> out like for like?
>
> A more headyweight option may be to implement code in the Nimbus HA
> cluster to forward certain admin commands to the leader (as every member of
> the cluster should know who that is). If that functionality existed we
> could then create a load balanced service that would fire the admin
> commands at any Nimbus node and not need to return the Leader address to
> the client at all.
>
> I would love to hear people's opinions on the above (hopefully there is
> already a solution to this).
>
> [1] https://minikube.sigs.k8s.io/docs/
> [2]
> https://github.com/apache/storm/blob/0859cd191715a09f5c84e781e3d0e3ca89e60654/storm-client/src/jvm/org/apache/storm/utils/NimbusClient.java#L222
>
> Cheers,
>
> Tom Cooper
> tomcooper.dev | @tomncooper <https://twitter.com/tomncooper>
>