You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Hong Liang Teoh (Jira)" <ji...@apache.org> on 2025/06/26 16:35:00 UTC
[jira] [Updated] (FLINK-37683) [Connector][AWS] SSO credentials fail because of shaded dependencies missing
[ https://issues.apache.org/jira/browse/FLINK-37683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Liang Teoh updated FLINK-37683:
------------------------------------
Fix Version/s: aws-connector-5.2.0
(was: aws-connector-5.1.0)
> [Connector][AWS] SSO credentials fail because of shaded dependencies missing
> ----------------------------------------------------------------------------
>
> Key: FLINK-37683
> URL: https://issues.apache.org/jira/browse/FLINK-37683
> Project: Flink
> Issue Type: Bug
> Components: Connectors / AWS
> Affects Versions: aws-connector-3.0.0, aws-connector-4.0.0, aws-connector-4.1.0, aws-connector-3.1.0, aws-connector-4.2.0, aws-connector-4.3.0, aws-connector-4.4.0, aws-connector-5.0.0, aws-connector-5.1.0
> Environment: * Apache Flink (any version using {{flink-connector-aws}} based on AWS SDK v2)
> * {{flink-connector-aws}} (using {{{}KinesisStreamsSource{}}})
> * Java application
> * AWS credentials configured via {{~/.aws/config}} using SSO profiles (e.g., {{{}sso-session{}}}, {{{}sso_account_id{}}}, {{{}sso_role_name{}}}).
> * Running locally via {{flink run}} or deploying a job using the shaded connector.
> Reporter: Ryan Despain
> Priority: Minor
> Labels: pull-request-available
> Fix For: aws-connector-5.2.0
>
>
> *Problem:* Users attempting to authenticate to AWS services (specifically Kinesis via {{{}KinesisStreamsSource{}}}) using AWS SSO profiles configured in {{~/.aws/config}} encounter a runtime error when using the standard shaded {{{}flink-connector-aws{}}}. The application fails to load credentials. This issue prevents users who rely on AWS SSO from using the connector.
> *Steps to Reproduce:*
> # Configure local AWS credentials using an SSO profile in {{{}~/.aws/config{}}}. Ensure the profile works with the AWS CLI.
> # Create a simple Flink Java application that uses {{KinesisStreamsSource}} to connect to an AWS Kinesis stream.
> # Include the {{org.apache.flink:flink-connector-aws:<version>}} dependency (which brings in the shaded AWS SDK).
> # Attempt to run the job using {{flink run}} or by building an application jar and running it.
> # _Expected Result:_ The job connects successfully using the SSO credentials.
> # _Actual Result:_ The job fails with an error related to loading credentials.
> _(Optional: Link to reproduction repository provided by the user: [soisyourface/flink-kinesis-blah|https://www.google.com/search?q=https://github.com/soisyourface/flink-kinesis-blah])_
> *Error Message:*
> {noformat}
> Unable to load credentials from any of the providers in the chain{noformat}
> And specifically:
> {noformat}
> To use Sso related properties in the 'default' profile, the 'sso' servicee module must be on the class path.{noformat}
>
> *Root Cause Analysis:* The AWS SDK for Java v2 uses {{software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils}} which dynamically loads the {{SSO_PROFILE_CREDENTIALS_PROVIDER_FACTORY}} using {{{}ClassLoaderHelper.loadClass{}}}.
> The {{flink-connector-aws}} project shades AWS SDK dependencies into {{flink-connector-aws-base}} to avoid classpath conflicts. However, the necessary AWS SDK modules for SSO functionality ({{{}software.amazon.awssdk:sso{}}} and {{{}software.amazon.awssdk:ssooidc{}}}) are not included as dependencies in the {{{}flink-connector-aws-base/pom.xml{}}}.
> Consequently, when Flink shades the dependencies, these modules are omitted from the final shaded jar. At runtime, the {{ProfileCredentialsUtils}} attempts to load the SSO factory class, but it cannot be found on the classpath because it wasn't included in the shaded artifact, leading to the observed error.
> *Proposed Solution:* Add {{software.amazon.awssdk:sso}} and {{software.amazon.awssdk:ssooidc}} as {{<dependency>}} entries within the {{<dependencies>}} section of the {{flink-connector-aws-base/pom.xml}} file in the {{apache/flink-connector-aws}} repository. This will ensure these modules are included when the connector is built and shaded.
> *Verification:* The user who reported this issue verified locally that building {{flink-connector-aws}} with these added dependencies resolved the problem in their Flink application.
> *Related PR:* A Pull Request with the proposed fix has been submitted: [apache/flink-connector-aws#197|https://www.google.com/search?q=https://github.com/apache/flink-connector-aws/pull/197]
> *Slack Conversation:* https://apache-flink.slack.com/archives/C03G7LJTS2G/p1743713205172909
--
This message was sent by Atlassian Jira
(v8.20.10#820010)