[I] Cross-account Kinesis source support via STS assume-role in JsonKinesisSource [hudi]

["suryadanny (via GitHub)" <gi...@apache.org>]

suryadanny opened a new issue, #19382:
URL: https://github.com/apache/hudi/issues/19382

   ## Problem
   
   Amazon Kinesis has no resource-based (resource) policy, so a Hudi ingestion job cannot read a Kinesis stream that lives in a **different AWS account** than the application — the default credential chain (the job's own pod/node/task role) gets `AccessDenied`. The only way to read cross-account is IAM role assumption (`sts:AssumeRole`).
   
   Today `JsonKinesisSource` / `KinesisOffsetGen` build the `KinesisClient` with region + the default credential chain (or static access/secret keys for LocalStack-style endpoints), with no way to assume a role in the stream's account.
   
   ## Proposed enhancement
   
   Add an optional config `hoodie.streamer.source.kinesis.role.arn`. When set, the Kinesis client is built with an auto-refreshing `StsAssumeRoleCredentialsProvider` (AWS SDK v2) for that role; the base STS client uses the default credential chain (which must be granted `sts:AssumeRole` on the ARN). When empty/absent, behavior is unchanged (same-account read via the default chain). Static access/secret keys continue to take precedence. Naming aligns with the existing `HoodieAWSConfig.hoodie.aws.role.arn`.
   
   The assume-role provider is cached per `region|roleArn` so a long-lived streaming executor reuses one `StsClient` per distinct role rather than leaking one per `mapPartitions` task.
   
   ## Environment
   - Component: hudi-utilities (DeltaStreamer/Streamer Kinesis source)
   
   A PR implementing this is attached.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Cross-account Kinesis source support via STS assume-role in JsonKinesisSource [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous closed issue #19382: Cross-account Kinesis source support via STS assume-role in JsonKinesisSource
URL: https://github.com/apache/hudi/issues/19382


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org