You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by XU Qinghui <qi...@gmail.com> on 2020/10/07 10:31:19 UTC

flink configuration: best practice for checkpoint storage secrets

Hello, folks

We are trying to use S3 for the checkpoint storage, and this involves some
secrets in the configuration. We tried two approaches to configure those
secrets:
- in the jvm application argument for jobmanager and taskmanager, such as
-Ds3.secret-key
- in the flink-conf.yaml file for jobmanager and taskmanager

Is there a third way? What's the best practice?
Thanks a lot!

Best regards,
Qinghui

Re: flink configuration: best practice for checkpoint storage secrets

Posted by XU Qinghui <qi...@gmail.com>.

Hello Till

Thanks a lot for the reply. But it turns out the IAM is applicable only
when the job is running inside AWS, which is not my case (basically we are
just using the S3 API provided by other services).
By reading again the flink doc, it seems it's suggesting to use the
flink-conf.yaml file, though.

Best regards,
Qinghui

Le mer. 7 oct. 2020 à 18:21, Till Rohrmann <tr...@apache.org> a écrit :

> Hi Qinghui,
>
> the recommended way would be to use AWS identity and access management
> (IAM) [1] if possible.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#configure-access-credentials
>
> Cheers,
> Till
>
> On Wed, Oct 7, 2020 at 12:31 PM XU Qinghui <qi...@gmail.com> wrote:
>
>> Hello, folks
>>
>> We are trying to use S3 for the checkpoint storage, and this
>> involves some secrets in the configuration. We tried two approaches to
>> configure those secrets:
>> - in the jvm application argument for jobmanager and taskmanager, such as
>> -Ds3.secret-key
>> - in the flink-conf.yaml file for jobmanager and taskmanager
>>
>> Is there a third way? What's the best practice?
>> Thanks a lot!
>>
>> Best regards,
>> Qinghui
>>
>

Re: flink configuration: best practice for checkpoint storage secrets

Posted by Till Rohrmann <tr...@apache.org>.

Hi Qinghui,

the recommended way would be to use AWS identity and access management
(IAM) [1] if possible.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#configure-access-credentials

Cheers,
Till

On Wed, Oct 7, 2020 at 12:31 PM XU Qinghui <qi...@gmail.com> wrote:

> Hello, folks
>
> We are trying to use S3 for the checkpoint storage, and this involves some
> secrets in the configuration. We tried two approaches to configure those
> secrets:
> - in the jvm application argument for jobmanager and taskmanager, such as
> -Ds3.secret-key
> - in the flink-conf.yaml file for jobmanager and taskmanager
>
> Is there a third way? What's the best practice?
> Thanks a lot!
>
> Best regards,
> Qinghui
>