Kubernetes - Configuring Datadog secret

[Daniel Fischer <d....@gmail.com>]

Hi,

I have a Flink application deployed to Kuberenetes and am trying to enable
Datadog metrics.  I'm stuck on how to set the
metrics.reporter.dghttp.apikey property
to a sensitive value.

I'm currently configuring my flink-conf.yaml file through a ConfigMap as
prescribed in the documentation. I don't want to hardcode the api key in
the ConfigMap, and actually I don't even want to inject it dynamically into
the ConfigMap because ConfigMaps aren't meant to hold secrets as I
understand it. I tried to set the property through
FLINK_PROPERTIES=metrics.reporter.dghttp.apikey:
<key>, but that doesn't seem to have any effect. It looks like
docker-entrypoint.sh
is attempting to read FLINK_PROPERTIES and inject them into the
flink-conf.yaml, but can't because flink-conf.yaml is mounted from the
ConfigMap and is therefore a read only file. I'm seeing this error in the
logs that supports that:

cannot move ‘/opt/flink/conf/flink-conf.yaml.tmp’ to
‘/opt/flink/conf/flink-conf.yaml’: Device or resource busy

/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Permission
denied

Any advice on how to set metrics.reporter.dghttp.apikey in a secure way?
Seems like I might need to move away from using a ConfigMap.

Thanks, Dan

Re: Kubernetes - Configuring Datadog secret

[Daniel Fischer <d....@gmail.com>]

Still open to thoughts + suggestions, but I ended up getting this working
by mounting the ConfigMap not to flink-conf.yaml but a different file (say
tmp-flink-conf.yaml); changing my docker image to use a custom entrypoint
script; and in that entrypoint script, copying tmp-flink-conf.yaml to
flink-conf.yaml
before calling through to /docker-entrypoint.sh. That way, flink-conf.yaml is
writable and /docker-entrypoint.sh can successfully write the contents of
FLINK_PROPERTIES to  flink-conf.yaml. (In my helm file I have FLINK_PROPERTIES
set to metrics.reporter.dghttp.apikey: $(DATADOG_API_KEY), with DATADOG_API_KEY
being injected through a SecretsProviderClass).

Dan

On Sun, Jul 31, 2022 at 9:56 AM Daniel Fischer <d....@gmail.com>
wrote:

> Hi,
>
> I have a Flink application deployed to Kuberenetes and am trying to enable
> Datadog metrics.  I'm stuck on how to set the
> metrics.reporter.dghttp.apikey property to a sensitive value.
>
> I'm currently configuring my flink-conf.yaml file through a ConfigMap as
> prescribed in the documentation. I don't want to hardcode the api key in
> the ConfigMap, and actually I don't even want to inject it dynamically into
> the ConfigMap because ConfigMaps aren't meant to hold secrets as I
> understand it. I tried to set the property through FLINK_PROPERTIES=metrics.reporter.dghttp.apikey:
> <key>, but that doesn't seem to have any effect. It looks like docker-entrypoint.sh
> is attempting to read FLINK_PROPERTIES and inject them into the
> flink-conf.yaml, but can't because flink-conf.yaml is mounted from the
> ConfigMap and is therefore a read only file. I'm seeing this error in the
> logs that supports that:
>
> cannot move ‘/opt/flink/conf/flink-conf.yaml.tmp’ to
> ‘/opt/flink/conf/flink-conf.yaml’: Device or resource busy
>
> /docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml:
> Permission denied
>
> Any advice on how to set metrics.reporter.dghttp.apikey in a secure way?
> Seems like I might need to move away from using a ConfigMap.
>
> Thanks, Dan
>