[GitHub] [beam] mosche commented on issue #23206: [New feature]: Support custom profile name in AWS ProfileCredentialsProvider

[GitBox <gi...@apache.org>]

mosche commented on issue #23206:
URL: https://github.com/apache/beam/issues/23206#issuecomment-1271636243

   > Isn't it a responsibility of users to check their configuration on workers and not Beam/KinesisIO? I mean why KinesisIO (or any other AWS IO connector) has to check if it's matching or not?
   
   Beam's responsibility is backwards compatibility in this case, @aromanenko-dev.
   If we add support for custom profile names for `ProfileCredentialsProvider` (will say we extract/serialize the profile name in the respective pipeline option), it gets a bit tricky as I tried to explain above. When serializing a provider that was set programmatically, we cannot distinguish if the user provided the profile name or if it was loaded from the system environment (e.g. `AWS_PROFILE=master`). If the latter is the case and the system environment on a worker node, where the credentials are eventually used, looks differently (e.g. `AWS_PROFILE=worker`) things can easily break for pipelines that used to run  successfully before: we would force the `master` profile on a worker even though it should use the `worker` profile there 💥 
   
   On the other hand, in case we should decide to not support this, it's still important to check the system environment during serialization of pipeline options. It's the only way to detect if the user set an unsupported ProfileCredentialsProvider programmatically with a custom profile name (`ProfileCredentialsProvider.create("myprofile")`). If we don't check it and  silently ignore the configuration (as done now), things break fairly randomly on workers as experienced here in this ticket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org