(tooling-gofannon) branch main updated: llm_service: force IMDS credentials for bedrock-mythos assume-role

[ak...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-gofannon.git


The following commit(s) were added to refs/heads/main by this push:
     new 19324e5  llm_service: force IMDS credentials for bedrock-mythos assume-role
19324e5 is described below

commit 19324e572090ade21c2f6072b500f54a6df28581
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Mon Jul 13 12:02:40 2026 -0700

    llm_service: force IMDS credentials for bedrock-mythos assume-role
    
    The container has AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY set to MinIO
    credentials for S3-compatible storage access. Boto3's default credential
    chain picks those up before falling through to the EC2 instance role,
    causing sts.assume_role() into the mythos account to send MinIO
    credentials to AWS and get InvalidClientTokenId.
    
    Force IMDS first via InstanceMetadataProvider. Falls back to the default
    chain (honoring AWS_PROFILE) in dev where IMDS is unreachable.
---
 .../api/user-service/services/llm_service.py       | 31 +++++++++++++++++-----
 1 file changed, 25 insertions(+), 6 deletions(-)

diff --git a/webapp/packages/api/user-service/services/llm_service.py b/webapp/packages/api/user-service/services/llm_service.py
index d66a7a1..dd3b40c 100644
--- a/webapp/packages/api/user-service/services/llm_service.py
+++ b/webapp/packages/api/user-service/services/llm_service.py
@@ -113,12 +113,31 @@ def _get_bedrock_mythos_api_key(
         if (cached["_expires_at"] - now).total_seconds() > _TOKEN_REFRESH_MARGIN_SECONDS:
             return cached["token"]
 
-    # Pick a session that ignores env-var credentials when AWS_PROFILE
-    # is set, so dev environments with minioadmin/etc. baked in don't
-    # poison the STS call. Default chain (EC2 instance role via IMDS)
-    # is used in prod where no profile is set.
-    profile = os.environ.get("AWS_PROFILE")
-    session = boto3.Session(profile_name=profile) if profile else boto3.Session()
+    # Force IMDS first (production). Falls back to default chain in dev
+    # where IMDS is unreachable and AWS_PROFILE-based SSO is expected.
+    # This bypasses the container's AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
+    # env vars which are set to MinIO credentials and would otherwise
+    # shadow the EC2 instance role.
+    from botocore.credentials import InstanceMetadataProvider
+    from botocore.utils import InstanceMetadataFetcher
+
+    session = None
+    try:
+        fetcher = InstanceMetadataFetcher(timeout=2, num_attempts=2)
+        provider = InstanceMetadataProvider(iam_role_fetcher=fetcher)
+        imds_creds = provider.load()
+        if imds_creds is not None:
+            session = boto3.Session(
+                aws_access_key_id=imds_creds.access_key,
+                aws_secret_access_key=imds_creds.secret_key,
+                aws_session_token=imds_creds.token,
+            )
+    except Exception:
+        pass  # IMDS unavailable; fall through
+
+    if session is None:
+        profile = os.environ.get("AWS_PROFILE")
+        session = boto3.Session(profile_name=profile) if profile else boto3.Session()
     sts = session.client("sts")
     response = sts.assume_role(
         RoleArn=assume_role_arn,


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org