You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Francis Conroy <fr...@switchdin.com> on 2022/04/28 09:30:46 UTC
Using the official flink operator and kubernetes secrets
Hi all,
I'm trying to use a kubernetes secret as a command line argument in my job
and the text replacement doesn't seem to be happening. I've verified
passing the custom args via the command line on my local flink cluster but
can't seem to get the environment var replacement to work.
apiVersion: flink.apache.org/v1alpha1
kind: FlinkDeployment
metadata:
namespace: default
name: http-over-mqtt
spec:
image: flink:1.14.4-scala_2.12-java11
flinkVersion: v1_14
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
kubernetes.env.secretKeyRef:
"env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
# containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
serviceAccount: flink
jobManager:
replicas: 1
resource:
memory: "1024m"
cpu: 1
taskManager:
resource:
memory: "1024m"
cpu: 1
podTemplate:
spec:
serviceAccount: flink
containers:
- name: flink-main-container
volumeMounts:
- mountPath: /flink-job
name: flink-jobs
env:
- name: DJANGO_TOKEN # kubectl create secret generic
switchdin-django-token --from-literal=token='[TOKEN]'
valueFrom:
secretKeyRef:
name: switchdin-django-token
key: token
optional: false
initContainers:
- name: grab-mqtt-over-http-jar
image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
command: [ "/bin/sh", "-c",
"cp /opt/switchdin/* /tmp/job/." ] # Copies the
jar in the init container to the flink-jobs volume
volumeMounts:
- name: flink-jobs
mountPath: /tmp/job
volumes:
- name: flink-jobs
emptyDir: { }
job:
jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
args: ["--swit-django-token", "$DJANGO_TOKEN",
"--swit-prod","false"]
entryClass: org.switchdin.HTTPOverMQTT
parallelism: 1
upgradeMode: stateless
state: running
In the logs I can see:
2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] -
--swit-django-token
2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - $DJANGO_TOKEN
2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
Anyone know how I can do this? I'm considering mounting it in a volume, but
that seems like a lot of hassle for such a small thing.
Thanks in advance!
--
This email and any attachments are proprietary and confidential and are
intended solely for the use of the individual to whom it is addressed. Any
views or opinions expressed are solely those of the author and do not
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
received this email in error, please let us know immediately by reply email
and delete it from your system. You may not use, disseminate, distribute or
copy this message nor disclose its contents to anyone.
SwitchDin Pty Ltd
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
Re: Using the official flink operator and kubernetes secrets
Posted by Yang Wang <da...@gmail.com>.
Thanks Meissner Dylan for the suggestion. I have created a ticket [1] to
track this requirement.
[1]. https://issues.apache.org/jira/browse/FLINK-27491
Best,
Yang
Francis Conroy <fr...@switchdin.com> 于2022年5月5日周四 06:06写道：
> Hi all,
>
> Thanks for looking into this. Yeah, I kept trying different variations of
> the replacement fields with no success. I'm trying to use the .getenv()
> technique now but our cluster is having problems and I haven't been able to
> reinstall the operator.
> I'll reply once it's all working.
>
> Thanks,
> Francis
>
> On Thu, 5 May 2022 at 03:23, Meissner, Dylan <
> dylan.t.meissner@nordstrom.com> wrote:
>
>> Flink deployment resources support env interpolation natively using $()
>> syntax. I expected this to "just work" like other resources when using the
>> operator, but it does not.
>>
>>
>> https://kubernetes.io/docs/tasks/inject-data-application/_print/#use-environment-variables-to-define-arguments
>>
>> job:
>> jarURI: local:///my.jar
>> entryClass: my.JobMainKt
>> args:
>> - "--kafka.bootstrap.servers"
>> - "my.kafka.host:9093"
>> - "--kafka.sasl.username"
>> - "$(KAFKA_SASL_USERNAME)"
>> - "--kafka.sasl.password"
>> - "$(KAFKA_SASL_PASSWORD)"
>>
>>
>> It would be a great addition, simplifying job startup decision-making
>> while following existing conventions.
>>
>> ------------------------------
>> *From:* Yang Wang <da...@gmail.com>
>> *Sent:* Tuesday, May 3, 2022 7:22 AM
>> *To:* Őrhidi Mátyás <ma...@gmail.com>
>> *Cc:* Francis Conroy <fr...@switchdin.com>; user <
>> user@flink.apache.org>
>> *Subject:* Re: Using the official flink operator and kubernetes secrets
>>
>> Flink could not support environment replacement in the args. I think you
>> could access the env via "*System.getenv()*" in the user main method.
>> It should work since the user main method is executed in the JobManager
>> side.
>>
>> Best,
>> Yang
>>
>> Őrhidi Mátyás <ma...@gmail.com> 于2022年4月28日周四 19:27写道：
>>
>> Also,
>>
>> just declaring it in the flink configs should be sufficient, no need to
>> define it in the pod templates:
>>
>> flinkConfiguration:
>> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
>>
>>
>> Best,
>> Matyas
>>
>> On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <ma...@gmail.com>
>> wrote:
>>
>> Hi Francis,
>>
>> I suggest accessing the environment variables directly, no need to pass
>> them as command arguments I guess.
>>
>> Best,
>> Matyas
>>
>> On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <
>> francis.conroy@switchdin.com> wrote:
>>
>> Hi all,
>>
>> I'm trying to use a kubernetes secret as a command line argument in my
>> job and the text replacement doesn't seem to be happening. I've verified
>> passing the custom args via the command line on my local flink cluster but
>> can't seem to get the environment var replacement to work.
>>
>> apiVersion: flink.apache.org/v1alpha1
>> kind: FlinkDeployment
>> metadata:
>> namespace: default
>> name: http-over-mqtt
>> spec:
>> image: flink:1.14.4-scala_2.12-java11
>> flinkVersion: v1_14
>> flinkConfiguration:
>> taskmanager.numberOfTaskSlots: "2"
>> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
>> # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
>> serviceAccount: flink
>> jobManager:
>> replicas: 1
>> resource:
>> memory: "1024m"
>> cpu: 1
>> taskManager:
>> resource:
>> memory: "1024m"
>> cpu: 1
>> podTemplate:
>> spec:
>> serviceAccount: flink
>> containers:
>> - name: flink-main-container
>> volumeMounts:
>> - mountPath: /flink-job
>> name: flink-jobs
>> env:
>> - name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]'
>> valueFrom:
>> secretKeyRef:
>> name: switchdin-django-token
>> key: token
>> optional: false
>> initContainers:
>> - name: grab-mqtt-over-http-jar
>> image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
>> command: [ "/bin/sh", "-c",
>> "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume
>> volumeMounts:
>> - name: flink-jobs
>> mountPath: /tmp/job
>> volumes:
>> - name: flink-jobs
>> emptyDir: { }
>> job:
>> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
>> args: ["--swit-django-token", "$DJANGO_TOKEN",
>> "--swit-prod","false"]
>> entryClass: org.switchdin.HTTPOverMQTT
>> parallelism: 1
>> upgradeMode: stateless
>> state: running
>>
>> In the logs I can see:
>>
>> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
>> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] -
>> --swit-django-token
>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] -
>> $DJANGO_TOKEN
>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
>>
>> Anyone know how I can do this? I'm considering mounting it in a volume,
>> but that seems like a lot of hassle for such a small thing.
>>
>> Thanks in advance!
>>
>>
>> This email and any attachments are proprietary and confidential and are
>> intended solely for the use of the individual to whom it is addressed. Any
>> views or opinions expressed are solely those of the author and do not
>> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
>> received this email in error, please let us know immediately by reply email
>> and delete it from your system. You may not use, disseminate, distribute or
>> copy this message nor disclose its contents to anyone.
>> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
>> Australia
>>
>>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>
Re: Using the official flink operator and kubernetes secrets
Posted by Francis Conroy <fr...@switchdin.com>.
Hi all,
Thanks for looking into this. Yeah, I kept trying different variations of
the replacement fields with no success. I'm trying to use the .getenv()
technique now but our cluster is having problems and I haven't been able to
reinstall the operator.
I'll reply once it's all working.
Thanks,
Francis
On Thu, 5 May 2022 at 03:23, Meissner, Dylan <dy...@nordstrom.com>
wrote:
> Flink deployment resources support env interpolation natively using $()
> syntax. I expected this to "just work" like other resources when using the
> operator, but it does not.
>
>
> https://kubernetes.io/docs/tasks/inject-data-application/_print/#use-environment-variables-to-define-arguments
>
> job:
> jarURI: local:///my.jar
> entryClass: my.JobMainKt
> args:
> - "--kafka.bootstrap.servers"
> - "my.kafka.host:9093"
> - "--kafka.sasl.username"
> - "$(KAFKA_SASL_USERNAME)"
> - "--kafka.sasl.password"
> - "$(KAFKA_SASL_PASSWORD)"
>
>
> It would be a great addition, simplifying job startup decision-making
> while following existing conventions.
>
> ------------------------------
> *From:* Yang Wang <da...@gmail.com>
> *Sent:* Tuesday, May 3, 2022 7:22 AM
> *To:* Őrhidi Mátyás <ma...@gmail.com>
> *Cc:* Francis Conroy <fr...@switchdin.com>; user <
> user@flink.apache.org>
> *Subject:* Re: Using the official flink operator and kubernetes secrets
>
> Flink could not support environment replacement in the args. I think you
> could access the env via "*System.getenv()*" in the user main method.
> It should work since the user main method is executed in the JobManager
> side.
>
> Best,
> Yang
>
> Őrhidi Mátyás <ma...@gmail.com> 于2022年4月28日周四 19:27写道：
>
> Also,
>
> just declaring it in the flink configs should be sufficient, no need to
> define it in the pod templates:
>
> flinkConfiguration:
> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
>
>
> Best,
> Matyas
>
> On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <ma...@gmail.com>
> wrote:
>
> Hi Francis,
>
> I suggest accessing the environment variables directly, no need to pass
> them as command arguments I guess.
>
> Best,
> Matyas
>
> On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <
> francis.conroy@switchdin.com> wrote:
>
> Hi all,
>
> I'm trying to use a kubernetes secret as a command line argument in my job
> and the text replacement doesn't seem to be happening. I've verified
> passing the custom args via the command line on my local flink cluster but
> can't seem to get the environment var replacement to work.
>
> apiVersion: flink.apache.org/v1alpha1
> kind: FlinkDeployment
> metadata:
> namespace: default
> name: http-over-mqtt
> spec:
> image: flink:1.14.4-scala_2.12-java11
> flinkVersion: v1_14
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
> # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
> serviceAccount: flink
> jobManager:
> replicas: 1
> resource:
> memory: "1024m"
> cpu: 1
> taskManager:
> resource:
> memory: "1024m"
> cpu: 1
> podTemplate:
> spec:
> serviceAccount: flink
> containers:
> - name: flink-main-container
> volumeMounts:
> - mountPath: /flink-job
> name: flink-jobs
> env:
> - name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]'
> valueFrom:
> secretKeyRef:
> name: switchdin-django-token
> key: token
> optional: false
> initContainers:
> - name: grab-mqtt-over-http-jar
> image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
> command: [ "/bin/sh", "-c",
> "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume
> volumeMounts:
> - name: flink-jobs
> mountPath: /tmp/job
> volumes:
> - name: flink-jobs
> emptyDir: { }
> job:
> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
> args: ["--swit-django-token", "$DJANGO_TOKEN",
> "--swit-prod","false"]
> entryClass: org.switchdin.HTTPOverMQTT
> parallelism: 1
> upgradeMode: stateless
> state: running
>
> In the logs I can see:
>
> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] -
> --swit-django-token
> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - $DJANGO_TOKEN
> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
>
> Anyone know how I can do this? I'm considering mounting it in a volume,
> but that seems like a lot of hassle for such a small thing.
>
> Thanks in advance!
>
>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>
>
--
This email and any attachments are proprietary and confidential and are
intended solely for the use of the individual to whom it is addressed. Any
views or opinions expressed are solely those of the author and do not
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
received this email in error, please let us know immediately by reply email
and delete it from your system. You may not use, disseminate, distribute or
copy this message nor disclose its contents to anyone.
SwitchDin Pty Ltd
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
Re: Using the official flink operator and kubernetes secrets
Posted by "Meissner, Dylan" <dy...@nordstrom.com>.
Flink deployment resources support env interpolation natively using $() syntax. I expected this to "just work" like other resources when using the operator, but it does not.
https://kubernetes.io/docs/tasks/inject-data-application/_print/#use-environment-variables-to-define-arguments
job:
jarURI: local:///my.jar
entryClass: my.JobMainKt
args:
- "--kafka.bootstrap.servers"
- "my.kafka.host:9093"
- "--kafka.sasl.username"
- "$(KAFKA_SASL_USERNAME)"
- "--kafka.sasl.password"
- "$(KAFKA_SASL_PASSWORD)"
It would be a great addition, simplifying job startup decision-making while following existing conventions.
________________________________
From: Yang Wang <da...@gmail.com>
Sent: Tuesday, May 3, 2022 7:22 AM
To: Őrhidi Mátyás <ma...@gmail.com>
Cc: Francis Conroy <fr...@switchdin.com>; user <us...@flink.apache.org>
Subject: Re: Using the official flink operator and kubernetes secrets
Flink could not support environment replacement in the args. I think you could access the env via "System.getenv()" in the user main method.
It should work since the user main method is executed in the JobManager side.
Best,
Yang
Őrhidi Mátyás <ma...@gmail.com>> 于2022年4月28日周四 19:27写道：
Also,
just declaring it in the flink configs should be sufficient, no need to define it in the pod templates:
flinkConfiguration:
kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
Best,
Matyas
On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <ma...@gmail.com>> wrote:
Hi Francis,
I suggest accessing the environment variables directly, no need to pass them as command arguments I guess.
Best,
Matyas
On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <fr...@switchdin.com>> wrote:
Hi all,
I'm trying to use a kubernetes secret as a command line argument in my job and the text replacement doesn't seem to be happening. I've verified passing the custom args via the command line on my local flink cluster but can't seem to get the environment var replacement to work.
apiVersion: flink.apache.org/v1alpha1<http://flink.apache.org/v1alpha1>
kind: FlinkDeployment
metadata:
namespace: default
name: http-over-mqtt
spec:
image: flink:1.14.4-scala_2.12-java11
flinkVersion: v1_14
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
# containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
serviceAccount: flink
jobManager:
replicas: 1
resource:
memory: "1024m"
cpu: 1
taskManager:
resource:
memory: "1024m"
cpu: 1
podTemplate:
spec:
serviceAccount: flink
containers:
- name: flink-main-container
volumeMounts:
- mountPath: /flink-job
name: flink-jobs
env:
- name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]'
valueFrom:
secretKeyRef:
name: switchdin-django-token
key: token
optional: false
initContainers:
- name: grab-mqtt-over-http-jar
image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
command: [ "/bin/sh", "-c",
"cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume
volumeMounts:
- name: flink-jobs
mountPath: /tmp/job
volumes:
- name: flink-jobs
emptyDir: { }
job:
jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
args: ["--swit-django-token", "$DJANGO_TOKEN",
"--swit-prod","false"]
entryClass: org.switchdin.HTTPOverMQTT
parallelism: 1
upgradeMode: stateless
state: running
In the logs I can see:
2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - --swit-django-token
2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - $DJANGO_TOKEN
2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
Anyone know how I can do this? I'm considering mounting it in a volume, but that seems like a lot of hassle for such a small thing.
Thanks in advance!
This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone.
SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
Re: Using the official flink operator and kubernetes secrets
Posted by Yang Wang <da...@gmail.com>.
Flink could not support environment replacement in the args. I think you
could access the env via "*System.getenv()*" in the user main method.
It should work since the user main method is executed in the JobManager
side.
Best,
Yang
Őrhidi Mátyás <ma...@gmail.com> 于2022年4月28日周四 19:27写道：
> Also,
>
> just declaring it in the flink configs should be sufficient, no need to
> define it in the pod templates:
>
> flinkConfiguration:
> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
>
>
> Best,
> Matyas
>
> On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <ma...@gmail.com>
> wrote:
>
>> Hi Francis,
>>
>> I suggest accessing the environment variables directly, no need to pass
>> them as command arguments I guess.
>>
>> Best,
>> Matyas
>>
>> On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <
>> francis.conroy@switchdin.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to use a kubernetes secret as a command line argument in my
>>> job and the text replacement doesn't seem to be happening. I've verified
>>> passing the custom args via the command line on my local flink cluster but
>>> can't seem to get the environment var replacement to work.
>>>
>>> apiVersion: flink.apache.org/v1alpha1
>>> kind: FlinkDeployment
>>> metadata:
>>> namespace: default
>>> name: http-over-mqtt
>>> spec:
>>> image: flink:1.14.4-scala_2.12-java11
>>> flinkVersion: v1_14
>>> flinkConfiguration:
>>> taskmanager.numberOfTaskSlots: "2"
>>> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
>>> # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
>>> serviceAccount: flink
>>> jobManager:
>>> replicas: 1
>>> resource:
>>> memory: "1024m"
>>> cpu: 1
>>> taskManager:
>>> resource:
>>> memory: "1024m"
>>> cpu: 1
>>> podTemplate:
>>> spec:
>>> serviceAccount: flink
>>> containers:
>>> - name: flink-main-container
>>> volumeMounts:
>>> - mountPath: /flink-job
>>> name: flink-jobs
>>> env:
>>> - name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]'
>>> valueFrom:
>>> secretKeyRef:
>>> name: switchdin-django-token
>>> key: token
>>> optional: false
>>> initContainers:
>>> - name: grab-mqtt-over-http-jar
>>> image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
>>> command: [ "/bin/sh", "-c",
>>> "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume
>>> volumeMounts:
>>> - name: flink-jobs
>>> mountPath: /tmp/job
>>> volumes:
>>> - name: flink-jobs
>>> emptyDir: { }
>>> job:
>>> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
>>> args: ["--swit-django-token", "$DJANGO_TOKEN",
>>> "--swit-prod","false"]
>>> entryClass: org.switchdin.HTTPOverMQTT
>>> parallelism: 1
>>> upgradeMode: stateless
>>> state: running
>>>
>>> In the logs I can see:
>>>
>>> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
>>> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] -
>>> --swit-django-token
>>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] -
>>> $DJANGO_TOKEN
>>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
>>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
>>>
>>> Anyone know how I can do this? I'm considering mounting it in a volume,
>>> but that seems like a lot of hassle for such a small thing.
>>>
>>> Thanks in advance!
>>>
>>>
>>> This email and any attachments are proprietary and confidential and are
>>> intended solely for the use of the individual to whom it is addressed. Any
>>> views or opinions expressed are solely those of the author and do not
>>> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
>>> received this email in error, please let us know immediately by reply email
>>> and delete it from your system. You may not use, disseminate, distribute or
>>> copy this message nor disclose its contents to anyone.
>>> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
>>> Australia
>>>
>>
Re: Using the official flink operator and kubernetes secrets
Posted by Őrhidi Mátyás <ma...@gmail.com>.
Also,
just declaring it in the flink configs should be sufficient, no need to
define it in the pod templates:
flinkConfiguration:
kubernetes.env.secretKeyRef:
"env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
Best,
Matyas
On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <ma...@gmail.com>
wrote:
> Hi Francis,
>
> I suggest accessing the environment variables directly, no need to pass
> them as command arguments I guess.
>
> Best,
> Matyas
>
> On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <
> francis.conroy@switchdin.com> wrote:
>
>> Hi all,
>>
>> I'm trying to use a kubernetes secret as a command line argument in my
>> job and the text replacement doesn't seem to be happening. I've verified
>> passing the custom args via the command line on my local flink cluster but
>> can't seem to get the environment var replacement to work.
>>
>> apiVersion: flink.apache.org/v1alpha1
>> kind: FlinkDeployment
>> metadata:
>> namespace: default
>> name: http-over-mqtt
>> spec:
>> image: flink:1.14.4-scala_2.12-java11
>> flinkVersion: v1_14
>> flinkConfiguration:
>> taskmanager.numberOfTaskSlots: "2"
>> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
>> # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
>> serviceAccount: flink
>> jobManager:
>> replicas: 1
>> resource:
>> memory: "1024m"
>> cpu: 1
>> taskManager:
>> resource:
>> memory: "1024m"
>> cpu: 1
>> podTemplate:
>> spec:
>> serviceAccount: flink
>> containers:
>> - name: flink-main-container
>> volumeMounts:
>> - mountPath: /flink-job
>> name: flink-jobs
>> env:
>> - name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]'
>> valueFrom:
>> secretKeyRef:
>> name: switchdin-django-token
>> key: token
>> optional: false
>> initContainers:
>> - name: grab-mqtt-over-http-jar
>> image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
>> command: [ "/bin/sh", "-c",
>> "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume
>> volumeMounts:
>> - name: flink-jobs
>> mountPath: /tmp/job
>> volumes:
>> - name: flink-jobs
>> emptyDir: { }
>> job:
>> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
>> args: ["--swit-django-token", "$DJANGO_TOKEN",
>> "--swit-prod","false"]
>> entryClass: org.switchdin.HTTPOverMQTT
>> parallelism: 1
>> upgradeMode: stateless
>> state: running
>>
>> In the logs I can see:
>>
>> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
>> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] -
>> --swit-django-token
>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] -
>> $DJANGO_TOKEN
>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
>> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
>>
>> Anyone know how I can do this? I'm considering mounting it in a volume,
>> but that seems like a lot of hassle for such a small thing.
>>
>> Thanks in advance!
>>
>>
>> This email and any attachments are proprietary and confidential and are
>> intended solely for the use of the individual to whom it is addressed. Any
>> views or opinions expressed are solely those of the author and do not
>> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
>> received this email in error, please let us know immediately by reply email
>> and delete it from your system. You may not use, disseminate, distribute or
>> copy this message nor disclose its contents to anyone.
>> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
>> Australia
>>
>
Re: Using the official flink operator and kubernetes secrets
Posted by Őrhidi Mátyás <ma...@gmail.com>.
Hi Francis,
I suggest accessing the environment variables directly, no need to pass
them as command arguments I guess.
Best,
Matyas
On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <
francis.conroy@switchdin.com> wrote:
> Hi all,
>
> I'm trying to use a kubernetes secret as a command line argument in my job
> and the text replacement doesn't seem to be happening. I've verified
> passing the custom args via the command line on my local flink cluster but
> can't seem to get the environment var replacement to work.
>
> apiVersion: flink.apache.org/v1alpha1
> kind: FlinkDeployment
> metadata:
> namespace: default
> name: http-over-mqtt
> spec:
> image: flink:1.14.4-scala_2.12-java11
> flinkVersion: v1_14
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token"
> # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN"
> serviceAccount: flink
> jobManager:
> replicas: 1
> resource:
> memory: "1024m"
> cpu: 1
> taskManager:
> resource:
> memory: "1024m"
> cpu: 1
> podTemplate:
> spec:
> serviceAccount: flink
> containers:
> - name: flink-main-container
> volumeMounts:
> - mountPath: /flink-job
> name: flink-jobs
> env:
> - name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]'
> valueFrom:
> secretKeyRef:
> name: switchdin-django-token
> key: token
> optional: false
> initContainers:
> - name: grab-mqtt-over-http-jar
> image: docker-push.k8s.local/test/switchdin/platform_flink:job-41
> command: [ "/bin/sh", "-c",
> "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume
> volumeMounts:
> - name: flink-jobs
> mountPath: /tmp/job
> volumes:
> - name: flink-jobs
> emptyDir: { }
> job:
> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
> args: ["--swit-django-token", "$DJANGO_TOKEN",
> "--swit-prod","false"]
> entryClass: org.switchdin.HTTPOverMQTT
> parallelism: 1
> upgradeMode: stateless
> state: running
>
> In the logs I can see:
>
> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {}
> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] -
> --swit-django-token
> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - $DJANGO_TOKEN
> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod
> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false
>
> Anyone know how I can do this? I'm considering mounting it in a volume,
> but that seems like a lot of hassle for such a small thing.
>
> Thanks in advance!
>
>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>