You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Edgar H <ka...@gmail.com> on 2022/08/23 12:53:37 UTC
Handling environment variables in Flink and within the Flink Operator
Morning all!
Recently we've been trying to migrate from some custom in-house code we've
got to manage our k8s cluster with Flink running into the recently released
operator.
However, we found an issue when parsing environment variables into the
arguments which are sent into the jobs - not being able to read those when
building that - https://issues.apache.org/jira/browse/FLINK-27491
Trying different workarounds we were thinking if environment variables in
the deployment could work but still, no luck so far.
So, my question is:
Which is the proper way to send environment variables into Flink jobs? Is
it even possible?
If it isn't, are there any plans on FLINK-27491 short-term?
Could it be possible to mount a file as a secret and read it from the job?
I assume we would be in the same situation as before and not even having
the file in the launched job.
Thanks!
Re: Handling environment variables in Flink and within the Flink Operator
Posted by Edgar H <ka...@gmail.com>.
Morning Francis,
Thanks a lot for the response!
Just tried but no luck so far, the Java code reading the env vars won't
catch it.
Are those envs promoted to jobs launched from the operator or ignored?
Thanks!
El mié., 24 ago. 2022 0:40, Francis Conroy <fr...@switchdin.com>
escribió:
> Hi Edgar,
>
> this is how we've been doing it and it's working well.
>
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> namespace: flink
> name: redacted-job
> spec:
> image: flink:1.15-java11
> flinkVersion: v1_15
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "1"
> kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:special-secret-token,key:token"
>
>
> Unfortunately, I don't think this is in the docs yet.
>
> Francis Conroy I Software Engineer
>
> Level 1, Building B, 91 Parry Street
>
> Newcastle NSW 2302
>
> *P* (02) 4786 0426 Ext 233
>
> E francis.conroy@switchdin.com
>
> www.switchdin.com
>
>
>
>
> On Tue, 23 Aug 2022 at 22:54, Edgar H <ka...@gmail.com> wrote:
>
>> Morning all!
>>
>> Recently we've been trying to migrate from some custom in-house code
>> we've got to manage our k8s cluster with Flink running into the recently
>> released operator.
>>
>> However, we found an issue when parsing environment variables into the
>> arguments which are sent into the jobs - not being able to read those when
>> building that - https://issues.apache.org/jira/browse/FLINK-27491
>>
>> Trying different workarounds we were thinking if environment variables in
>> the deployment could work but still, no luck so far.
>>
>> So, my question is:
>>
>> Which is the proper way to send environment variables into Flink jobs? Is
>> it even possible?
>> If it isn't, are there any plans on FLINK-27491 short-term?
>>
>> Could it be possible to mount a file as a secret and read it from the
>> job? I assume we would be in the same situation as before and not even
>> having the file in the launched job.
>>
>> Thanks!
>>
>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>
Re: Handling environment variables in Flink and within the Flink Operator
Posted by Francis Conroy <fr...@switchdin.com>.
Hi Edgar,
this is how we've been doing it and it's working well.
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
namespace: flink
name: redacted-job
spec:
image: flink:1.15-java11
flinkVersion: v1_15
flinkConfiguration:
taskmanager.numberOfTaskSlots: "1"
kubernetes.env.secretKeyRef:
"env:DJANGO_TOKEN,secret:special-secret-token,key:token"
Unfortunately, I don't think this is in the docs yet.
Francis Conroy I Software Engineer
Level 1, Building B, 91 Parry Street
Newcastle NSW 2302
*P* (02) 4786 0426 Ext 233
E francis.conroy@switchdin.com
www.switchdin.com
On Tue, 23 Aug 2022 at 22:54, Edgar H <ka...@gmail.com> wrote:
> Morning all!
>
> Recently we've been trying to migrate from some custom in-house code we've
> got to manage our k8s cluster with Flink running into the recently released
> operator.
>
> However, we found an issue when parsing environment variables into the
> arguments which are sent into the jobs - not being able to read those when
> building that - https://issues.apache.org/jira/browse/FLINK-27491
>
> Trying different workarounds we were thinking if environment variables in
> the deployment could work but still, no luck so far.
>
> So, my question is:
>
> Which is the proper way to send environment variables into Flink jobs? Is
> it even possible?
> If it isn't, are there any plans on FLINK-27491 short-term?
>
> Could it be possible to mount a file as a secret and read it from the job?
> I assume we would be in the same situation as before and not even having
> the file in the launched job.
>
> Thanks!
>
--
This email and any attachments are proprietary and confidential and are
intended solely for the use of the individual to whom it is addressed. Any
views or opinions expressed are solely those of the author and do not
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
received this email in error, please let us know immediately by reply email
and delete it from your system. You may not use, disseminate, distribute or
copy this message nor disclose its contents to anyone.
SwitchDin Pty Ltd
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia