You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2023/01/09 22:17:00 UTC
Possibilities for fetching config information from Kubernetes
All,
I'm aware that there is a k8s manager for clustering
(CloudMembershipService) but I was wondering if / how that could be
extended in order to provide any other types of automated configuration
information for a Tomcat installation.
For example, I'd love to be able to deploy a Tomcat node and have it
grab its primary database connection information from k8s.
I spent like 5 minutes reading through the CloudMembershipService and
KubernetesMembershipProvider classes and it seems to be all bundled
together and very geared toward fetching cluster information.
It seems that most of the KubernetesMembershipProvider.start method
could be moved into a separate support class which just manages k8s
connection information (e.g. fetching from the environment, building
URLs to various interesting paths, wrapping fethcing-and-JSON-parsing,
etc.) and that could be re-used for a parameter-resolver for XML config
files like we can do for system properties like
<Connector jvmRoute="${foo}">
It occurs to be that we should be able to do something like this:
<Connector jvmRoute="${k8s:jvmRoute}">
... and write a resolver that fetches that value on the fly. (Hopefully
it would cache stuff, so that a dozen different environmental references
don't have to be resolved separately.)
Does that sound useful to anyone? I've never used k8s but I'm looking at
making a service of mine easier to deploy in a totally automated way,
and this kind of thing would certainly help with that.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Possibilities for fetching config information from Kubernetes
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark, Rémy,
On 1/10/23 09:58, Rémy Maucherat wrote:
> On Tue, Jan 10, 2023 at 3:11 PM Christopher Schultz
> <ch...@christopherschultz.net> wrote:
>>
>> Mark,
>>
>> On 1/10/23 03:22, Mark Thomas wrote:
>>> On 09/01/2023 22:17, Christopher Schultz wrote:
>>>> All,
>>>>
>>>> I'm aware that there is a k8s manager for clustering
>>>> (CloudMembershipService) but I was wondering if / how that could be
>>>> extended in order to provide any other types of automated
>>>> configuration information for a Tomcat installation.
>>>>
>>>> For example, I'd love to be able to deploy a Tomcat node and have it
>>>> grab its primary database connection information from k8s.
>>>>
>>>> I spent like 5 minutes reading through the CloudMembershipService and
>>>> KubernetesMembershipProvider classes and it seems to be all bundled
>>>> together and very geared toward fetching cluster information.
>>>>
>>>> It seems that most of the KubernetesMembershipProvider.start method
>>>> could be moved into a separate support class which just manages k8s
>>>> connection information (e.g. fetching from the environment, building
>>>> URLs to various interesting paths, wrapping fethcing-and-JSON-parsing,
>>>> etc.) and that could be re-used for a parameter-resolver for XML
>>>> config files like we can do for system properties like
>>>>
>>>> <Connector jvmRoute="${foo}">
>>>>
>>>> It occurs to be that we should be able to do something like this:
>>>>
>>>> <Connector jvmRoute="${k8s:jvmRoute}">
>>>>
>>>> ... and write a resolver that fetches that value on the fly.
>>>> (Hopefully it would cache stuff, so that a dozen different
>>>> environmental references don't have to be resolved separately.)
>>>
>>> Isn't this why ServiceBindingPropertySource was contributed? I only
>>> dabble with k8s very occasionally but my understanding is that this is
>>> the 'proper' way to pick up config in a k8s environment.
>>
>> Yeah, I think you are right. I hadn't noticed that class before, and it
>> looks like it doesn't interact with k8s (or similar) in any way: it just
>> assumes that the orchestration framework has already dropped a cache of
>> useful configuration files onto the system before the service is launched.
>>
>> It looks like the KubernetesMembershipProvider needs to be separate from
>> that because it actually has to communicate directly with k8s for, well,
>> membership services. It's not just reading configuration values.
>>
>> So, I guess... nothing to see here!
>
> The KubernetesMembershipProvider is the cleanest way to access the
> Kube namespace. Then you can use it to list the pods (as here) or
> pretty much anything else. There's a problem though: it needs the
> token to connect to the service account (= configuration !) *and*
> configure the appropriate permissions in Kube to allow the desired
> access (= annoying configuration !). I liked it because once it works
> it is predictable and reliable.
>
> DNSMembershipProvider is an equivalent to get the list of pods with
> far less configuration. However it uses DNS which is prone to
> "issues".
>
> So basically yeah, you would need to write a PropertySource that does
> somewhat the connection and json parsing that
> KubernetesMembershipProvider does. I don't see how the processing can
> be generic for everything that is in the Kube namespace, so it would
> allow access to some specific values. The problem then is that
> security configuration (in Kube) will still be horrible. But it could
> work eventually.
I think ServiceBindingPropertySource already covers this, as long as you
use the servicebinding.io spec, which it seems k8s does.
> In KubernetesMembershipProvider, the connection with the token to the
> service account is generic. The URL used in the Kube namespace is
> hardcoded. And the json parsing is obviously specific to processing
> the list of (running) pods in the cluster.
Some of these items could even be parameterized using
ServiceBindingPropertySource -- same thing with grabbing the config to
drive KubernetesMembershipProvider, right?
It looks like if I wanted to use k8s for deployment of a simple JDBC
configuration, I could do something like this:
<Context>
<Resource [blah blah]
url="${app-database.url}&truststore=${SERVICE_BINDING_ROOT}/truststore.p12"
username="${app-database.username}"
password="${app-database.password}"
/>
</Context>
And for an SSL configuration (right from the
ServiceBindingPropertySource javadoc):
<SSLCHostConfig>
<Certificate certificateKeyFile="${SERVICE_BINDING_ROOT}/host.key"
certificateFile="${SERVICE_BINDING_ROOT}/host.crt" />
</SSLHostConfig>
I'm curious as to why the sample in ServiceBindingPropertySource javadoc
actually uses "service resources" for those values when I think it
doesn't make any sense. ServiceBindingPropertySource.getProperty() will
resolve the value of the property to the CONTENTS of the file, not to
the filename itself. This is what the javadoc says:
* <SSLHostConfig>
* <Certificate
certificateKeyFile="${custom-certificate.keyFile}"
* certificateFile="${custom-certificate.file}"
*
certificateChainFile="${custom-certificate.chainFile}"
* type="RSA" />
I think having certificateFile="-----BEGIN CERTIFICATE-----...." is just
not going to work, will it? I think this javadoc is wrong.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Possibilities for fetching config information from Kubernetes
Posted by Rémy Maucherat <re...@apache.org>.
On Tue, Jan 10, 2023 at 3:11 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> Mark,
>
> On 1/10/23 03:22, Mark Thomas wrote:
> > On 09/01/2023 22:17, Christopher Schultz wrote:
> >> All,
> >>
> >> I'm aware that there is a k8s manager for clustering
> >> (CloudMembershipService) but I was wondering if / how that could be
> >> extended in order to provide any other types of automated
> >> configuration information for a Tomcat installation.
> >>
> >> For example, I'd love to be able to deploy a Tomcat node and have it
> >> grab its primary database connection information from k8s.
> >>
> >> I spent like 5 minutes reading through the CloudMembershipService and
> >> KubernetesMembershipProvider classes and it seems to be all bundled
> >> together and very geared toward fetching cluster information.
> >>
> >> It seems that most of the KubernetesMembershipProvider.start method
> >> could be moved into a separate support class which just manages k8s
> >> connection information (e.g. fetching from the environment, building
> >> URLs to various interesting paths, wrapping fethcing-and-JSON-parsing,
> >> etc.) and that could be re-used for a parameter-resolver for XML
> >> config files like we can do for system properties like
> >>
> >> <Connector jvmRoute="${foo}">
> >>
> >> It occurs to be that we should be able to do something like this:
> >>
> >> <Connector jvmRoute="${k8s:jvmRoute}">
> >>
> >> ... and write a resolver that fetches that value on the fly.
> >> (Hopefully it would cache stuff, so that a dozen different
> >> environmental references don't have to be resolved separately.)
> >
> > Isn't this why ServiceBindingPropertySource was contributed? I only
> > dabble with k8s very occasionally but my understanding is that this is
> > the 'proper' way to pick up config in a k8s environment.
>
> Yeah, I think you are right. I hadn't noticed that class before, and it
> looks like it doesn't interact with k8s (or similar) in any way: it just
> assumes that the orchestration framework has already dropped a cache of
> useful configuration files onto the system before the service is launched.
>
> It looks like the KubernetesMembershipProvider needs to be separate from
> that because it actually has to communicate directly with k8s for, well,
> membership services. It's not just reading configuration values.
>
> So, I guess... nothing to see here!
The KubernetesMembershipProvider is the cleanest way to access the
Kube namespace. Then you can use it to list the pods (as here) or
pretty much anything else. There's a problem though: it needs the
token to connect to the service account (= configuration !) *and*
configure the appropriate permissions in Kube to allow the desired
access (= annoying configuration !). I liked it because once it works
it is predictable and reliable.
DNSMembershipProvider is an equivalent to get the list of pods with
far less configuration. However it uses DNS which is prone to
"issues".
So basically yeah, you would need to write a PropertySource that does
somewhat the connection and json parsing that
KubernetesMembershipProvider does. I don't see how the processing can
be generic for everything that is in the Kube namespace, so it would
allow access to some specific values. The problem then is that
security configuration (in Kube) will still be horrible. But it could
work eventually.
In KubernetesMembershipProvider, the connection with the token to the
service account is generic. The URL used in the Kube namespace is
hardcoded. And the json parsing is obviously specific to processing
the list of (running) pods in the cluster.
Rémy
> Thanks,
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Possibilities for fetching config information from Kubernetes
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 1/10/23 03:22, Mark Thomas wrote:
> On 09/01/2023 22:17, Christopher Schultz wrote:
>> All,
>>
>> I'm aware that there is a k8s manager for clustering
>> (CloudMembershipService) but I was wondering if / how that could be
>> extended in order to provide any other types of automated
>> configuration information for a Tomcat installation.
>>
>> For example, I'd love to be able to deploy a Tomcat node and have it
>> grab its primary database connection information from k8s.
>>
>> I spent like 5 minutes reading through the CloudMembershipService and
>> KubernetesMembershipProvider classes and it seems to be all bundled
>> together and very geared toward fetching cluster information.
>>
>> It seems that most of the KubernetesMembershipProvider.start method
>> could be moved into a separate support class which just manages k8s
>> connection information (e.g. fetching from the environment, building
>> URLs to various interesting paths, wrapping fethcing-and-JSON-parsing,
>> etc.) and that could be re-used for a parameter-resolver for XML
>> config files like we can do for system properties like
>>
>> <Connector jvmRoute="${foo}">
>>
>> It occurs to be that we should be able to do something like this:
>>
>> <Connector jvmRoute="${k8s:jvmRoute}">
>>
>> ... and write a resolver that fetches that value on the fly.
>> (Hopefully it would cache stuff, so that a dozen different
>> environmental references don't have to be resolved separately.)
>
> Isn't this why ServiceBindingPropertySource was contributed? I only
> dabble with k8s very occasionally but my understanding is that this is
> the 'proper' way to pick up config in a k8s environment.
Yeah, I think you are right. I hadn't noticed that class before, and it
looks like it doesn't interact with k8s (or similar) in any way: it just
assumes that the orchestration framework has already dropped a cache of
useful configuration files onto the system before the service is launched.
It looks like the KubernetesMembershipProvider needs to be separate from
that because it actually has to communicate directly with k8s for, well,
membership services. It's not just reading configuration values.
So, I guess... nothing to see here!
Thanks,
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Possibilities for fetching config information from Kubernetes
Posted by Mark Thomas <ma...@apache.org>.
On 09/01/2023 22:17, Christopher Schultz wrote:
> All,
>
> I'm aware that there is a k8s manager for clustering
> (CloudMembershipService) but I was wondering if / how that could be
> extended in order to provide any other types of automated configuration
> information for a Tomcat installation.
>
> For example, I'd love to be able to deploy a Tomcat node and have it
> grab its primary database connection information from k8s.
>
> I spent like 5 minutes reading through the CloudMembershipService and
> KubernetesMembershipProvider classes and it seems to be all bundled
> together and very geared toward fetching cluster information.
>
> It seems that most of the KubernetesMembershipProvider.start method
> could be moved into a separate support class which just manages k8s
> connection information (e.g. fetching from the environment, building
> URLs to various interesting paths, wrapping fethcing-and-JSON-parsing,
> etc.) and that could be re-used for a parameter-resolver for XML config
> files like we can do for system properties like
>
> <Connector jvmRoute="${foo}">
>
> It occurs to be that we should be able to do something like this:
>
> <Connector jvmRoute="${k8s:jvmRoute}">
>
> ... and write a resolver that fetches that value on the fly. (Hopefully
> it would cache stuff, so that a dozen different environmental references
> don't have to be resolved separately.)
Isn't this why ServiceBindingPropertySource was contributed? I only
dabble with k8s very occasionally but my understanding is that this is
the 'proper' way to pick up config in a k8s environment.
Mark
>
> Does that sound useful to anyone? I've never used k8s but I'm looking at
> making a service of mine easier to deploy in a totally automated way,
> and this kind of thing would certainly help with that.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org