You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Hong Liang Teoh (Jira)" <ji...@apache.org> on 2025/06/26 16:35:00 UTC

[jira] [Updated] (FLINK-37683) [Connector][AWS] SSO credentials fail because of shaded dependencies missing

     [ https://issues.apache.org/jira/browse/FLINK-37683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Liang Teoh updated FLINK-37683:
------------------------------------
    Fix Version/s: aws-connector-5.2.0
                       (was: aws-connector-5.1.0)

> [Connector][AWS] SSO credentials fail because of shaded dependencies missing
> ----------------------------------------------------------------------------
>
>                 Key: FLINK-37683
>                 URL: https://issues.apache.org/jira/browse/FLINK-37683
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / AWS
>    Affects Versions: aws-connector-3.0.0, aws-connector-4.0.0, aws-connector-4.1.0, aws-connector-3.1.0, aws-connector-4.2.0, aws-connector-4.3.0, aws-connector-4.4.0, aws-connector-5.0.0, aws-connector-5.1.0
>         Environment: * Apache Flink (any version using {{flink-connector-aws}} based on AWS SDK v2)
>  * {{flink-connector-aws}} (using {{{}KinesisStreamsSource{}}})
>  * Java application
>  * AWS credentials configured via {{~/.aws/config}} using SSO profiles (e.g., {{{}sso-session{}}}, {{{}sso_account_id{}}}, {{{}sso_role_name{}}}).
>  * Running locally via {{flink run}} or deploying a job using the shaded connector.
>            Reporter: Ryan Despain
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: aws-connector-5.2.0
>
>
> *Problem:* Users attempting to authenticate to AWS services (specifically Kinesis via {{{}KinesisStreamsSource{}}}) using AWS SSO profiles configured in {{~/.aws/config}} encounter a runtime error when using the standard shaded {{{}flink-connector-aws{}}}. The application fails to load credentials. This issue prevents users who rely on AWS SSO from using the connector.
> *Steps to Reproduce:*
>  # Configure local AWS credentials using an SSO profile in {{{}~/.aws/config{}}}. Ensure the profile works with the AWS CLI.
>  # Create a simple Flink Java application that uses {{KinesisStreamsSource}} to connect to an AWS Kinesis stream.
>  # Include the {{org.apache.flink:flink-connector-aws:<version>}} dependency (which brings in the shaded AWS SDK).
>  # Attempt to run the job using {{flink run}} or by building an application jar and running it.
>  # _Expected Result:_ The job connects successfully using the SSO credentials.
>  # _Actual Result:_ The job fails with an error related to loading credentials.
> _(Optional: Link to reproduction repository provided by the user: [soisyourface/flink-kinesis-blah|https://www.google.com/search?q=https://github.com/soisyourface/flink-kinesis-blah])_
> *Error Message:*
> {noformat}
> Unable to load credentials from any of the providers in the chain{noformat}
> And specifically:
> {noformat}
> To use Sso related properties in the 'default' profile, the 'sso' servicee module must be on the class path.{noformat}
>  
> *Root Cause Analysis:* The AWS SDK for Java v2 uses {{software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils}} which dynamically loads the {{SSO_PROFILE_CREDENTIALS_PROVIDER_FACTORY}} using {{{}ClassLoaderHelper.loadClass{}}}.
> The {{flink-connector-aws}} project shades AWS SDK dependencies into {{flink-connector-aws-base}} to avoid classpath conflicts. However, the necessary AWS SDK modules for SSO functionality ({{{}software.amazon.awssdk:sso{}}} and {{{}software.amazon.awssdk:ssooidc{}}}) are not included as dependencies in the {{{}flink-connector-aws-base/pom.xml{}}}.
> Consequently, when Flink shades the dependencies, these modules are omitted from the final shaded jar. At runtime, the {{ProfileCredentialsUtils}} attempts to load the SSO factory class, but it cannot be found on the classpath because it wasn't included in the shaded artifact, leading to the observed error.
> *Proposed Solution:* Add {{software.amazon.awssdk:sso}} and {{software.amazon.awssdk:ssooidc}} as {{<dependency>}} entries within the {{<dependencies>}} section of the {{flink-connector-aws-base/pom.xml}} file in the {{apache/flink-connector-aws}} repository. This will ensure these modules are included when the connector is built and shaded.
> *Verification:* The user who reported this issue verified locally that building {{flink-connector-aws}} with these added dependencies resolved the problem in their Flink application.
> *Related PR:* A Pull Request with the proposed fix has been submitted: [apache/flink-connector-aws#197|https://www.google.com/search?q=https://github.com/apache/flink-connector-aws/pull/197]
> *Slack Conversation:* https://apache-flink.slack.com/archives/C03G7LJTS2G/p1743713205172909



--
This message was sent by Atlassian Jira
(v8.20.10#820010)