[jira] [Commented] (CAMEL-24197) camel-aws2-kinesis - Fixed-shardId DescribeStream on every poll and batch partition key

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-24197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097251#comment-18097251 ] 

Claus Ibsen commented on CAMEL-24197:
-------------------------------------

Fixed via https://github.com/apache/camel/pull/24878 (merged to main).

_Claude Code on behalf of davsclaus_

> camel-aws2-kinesis - Fixed-shardId DescribeStream on every poll and batch partition key
> ---------------------------------------------------------------------------------------
>
>                 Key: CAMEL-24197
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24197
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-aws
>            Reporter: Claus Ibsen
>            Assignee: Claus Ibsen
>            Priority: Minor
>              Labels: review-2026-07
>             Fix For: 4.22.0
>
>
> Split from CAMEL-24157 (medium-severity findings, group 3).
> Two remaining Kinesis findings:
> *1. Fixed-shardId consumer calls DescribeStream on every poll*
> {{Kinesis2Consumer:108-141}} — When {{shardId}} is configured (non-empty), every call to {{poll()}} issues a {{DescribeStream}} API call to locate the Shard object. Two problems:
> - Rate-limit risk: DescribeStream has a 10 TPS per-account limit. With default 1-second poll delay, a single consumer approaches the limit; multiple consumers exceed it.
> - >100 shard failure: DescribeStream returns at most 100 shards per response. The code does not paginate (no {{exclusiveStartShardId}} loop), so for streams with >100 shards the configured shard may not be found, throwing {{IllegalStateException}}.
> The multi-shard path correctly uses a cached {{ShardMonitor}} with the {{ListShards}} API (paginated, higher rate limit). The fixed-shardId path bypasses this caching entirely.
> *2. Batch producer applies single partition key to all records*
> {{Kinesis2Producer:77-122}} — A single {{CamelAwsKinesisPartitionKey}} header is applied to every {{PutRecordsRequestEntry}} in the batch. All records are routed to the same shard. This is a design limitation — no per-record partition key mechanism exists in the current API.
> *Fix approach:*
> - #1: Cache the shard iterator for the configured shardId (similar to ShardMonitor), avoiding DescribeStream on every poll. Add pagination for >100-shard streams.
> - #2: Design decision needed — consider accepting {{List<PutRecordsRequestEntry>}} directly, or a wrapper type carrying per-record keys, or deriving partition keys from each record body.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)