You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Matt Moore <me...@mattdoescode.com> on 2018/08/08 17:54:31 UTC

Using sensitive configuration/credentials 

I'm wondering what the best practice is for using secrets in a Flink program, and I can't find any info in the docs or posted anywhere else.

I need to store an access token to one of my APIs for flink to use to dump results into, and right now I'm passing it through as a configuration parameter, but that doesn't seem like the most secure thing to do and the value shows up in the Flink Dashboard under Configuration which is less than ideal.

Has anyone else dealt with a situation like this?

Thanks,

Re: Using sensitive configuration/credentials

Posted by vino yang <ya...@gmail.com>.
Hi Chesnay,

Oh, I did not know this feature. Any more description in Flink official
documentation?

Thanks, vino.

Chesnay Schepler <ch...@apache.org> 于2018年8月9日周四 下午4:29写道：

> If you change the name of your configuration key ti include "secret" or
> "password" it should be hidden from the logs and UI.
>
> On 09.08.2018 04:28, vino yang wrote:
>
> Hi Matt,
>
> Flink is currently enhancing its security, such as the current data
> transmission can be configured with SSL mode[1].
> However, some problems involving configuration and web ui display do
> exist, and they are still displayed in plain text.
> I think a temporary way to do this is to keep your secret configuration in
> encrypted form elsewhere, such as Zookeeper or RDBMS, and then dynamically
> read it into the job in a UDF (in the open method).
>
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/security-ssl.html
>
> Thanks, vino.
>
> Matt Moore <me...@mattdoescode.com> 于2018年8月9日周四 上午1:54写道：
>
>> I'm wondering what the best practice is for using secrets in a Flink
>> program, and I can't find any info in the docs or posted anywhere else.
>>
>> I need to store an access token to one of my APIs for flink to use to
>> dump results into, and right now I'm passing it through as a configuration
>> parameter, but that doesn't seem like the most secure thing to do and the
>> value shows up in the Flink Dashboard under Configuration which is less
>> than ideal.
>>
>> Has anyone else dealt with a situation like this?
>>
>> Thanks,
>>
>>
>

Re: Using sensitive configuration/credentials

Posted by Chesnay Schepler <ch...@apache.org>.
If you change the name of your configuration key ti include "secret" or 
"password" it should be hidden from the logs and UI.

On 09.08.2018 04:28, vino yang wrote:
> Hi Matt,
>
> Flink is currently enhancing its security, such as the current data 
> transmission can be configured with SSL mode[1].
> However, some problems involving configuration and web ui display do 
> exist, and they are still displayed in plain text.
> I think a temporary way to do this is to keep your secret 
> configuration in encrypted form elsewhere, such as Zookeeper or RDBMS, 
> and then dynamically read it into the job in a UDF (in the open method).
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/security-ssl.html
>
> Thanks, vino.
>
> Matt Moore <me@mattdoescode.com <ma...@mattdoescode.com>> 
> 于2018年8月9日周四 上午1:54写道：
>
>     I'm wondering what the best practice is for using secrets in a
>     Flink program, and I can't find any info in the docs or posted
>     anywhere else.
>
>     I need to store an access token to one of my APIs for flink to use
>     to dump results into, and right now I'm passing it through as a
>     configuration parameter, but that doesn't seem like the most
>     secure thing to do and the value shows up in the Flink Dashboard
>     under Configuration which is less than ideal.
>
>     Has anyone else dealt with a situation like this?
>
>     Thanks,
>

Re: Using sensitive configuration/credentials

Posted by vino yang <ya...@gmail.com>.
Hi Matt,

Flink is currently enhancing its security, such as the current data
transmission can be configured with SSL mode[1].
However, some problems involving configuration and web ui display do exist,
and they are still displayed in plain text.
I think a temporary way to do this is to keep your secret configuration in
encrypted form elsewhere, such as Zookeeper or RDBMS, and then dynamically
read it into the job in a UDF (in the open method).

https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/security-ssl.html

Thanks, vino.

Matt Moore <me...@mattdoescode.com> 于2018年8月9日周四 上午1:54写道：

> I'm wondering what the best practice is for using secrets in a Flink
> program, and I can't find any info in the docs or posted anywhere else.
>
> I need to store an access token to one of my APIs for flink to use to dump
> results into, and right now I'm passing it through as a configuration
> parameter, but that doesn't seem like the most secure thing to do and the
> value shows up in the Flink Dashboard under Configuration which is less
> than ideal.
>
> Has anyone else dealt with a situation like this?
>
> Thanks,
>
>