[jira] [Created] (CAMEL-24197) camel-aws2-kinesis - Fixed-shardId DescribeStream on every poll and batch partition key

["Claus Ibsen (Jira)" <ji...@apache.org>]

Claus Ibsen created CAMEL-24197:
-----------------------------------

             Summary: camel-aws2-kinesis - Fixed-shardId DescribeStream on every poll and batch partition key
                 Key: CAMEL-24197
                 URL: https://issues.apache.org/jira/browse/CAMEL-24197
             Project: Camel
          Issue Type: Bug
          Components: camel-aws
            Reporter: Claus Ibsen


Split from CAMEL-24157 (medium-severity findings, group 3).

Two remaining Kinesis findings:

*1. Fixed-shardId consumer calls DescribeStream on every poll*

{{Kinesis2Consumer:108-141}} — When {{shardId}} is configured (non-empty), every call to {{poll()}} issues a {{DescribeStream}} API call to locate the Shard object. Two problems:
- Rate-limit risk: DescribeStream has a 10 TPS per-account limit. With default 1-second poll delay, a single consumer approaches the limit; multiple consumers exceed it.
- >100 shard failure: DescribeStream returns at most 100 shards per response. The code does not paginate (no {{exclusiveStartShardId}} loop), so for streams with >100 shards the configured shard may not be found, throwing {{IllegalStateException}}.

The multi-shard path correctly uses a cached {{ShardMonitor}} with the {{ListShards}} API (paginated, higher rate limit). The fixed-shardId path bypasses this caching entirely.

*2. Batch producer applies single partition key to all records*

{{Kinesis2Producer:77-122}} — A single {{CamelAwsKinesisPartitionKey}} header is applied to every {{PutRecordsRequestEntry}} in the batch. All records are routed to the same shard. This is a design limitation — no per-record partition key mechanism exists in the current API.

*Fix approach:*
- #1: Cache the shard iterator for the configured shardId (similar to ShardMonitor), avoiding DescribeStream on every poll. Add pagination for >100-shard streams.
- #2: Design decision needed — consider accepting {{List<PutRecordsRequestEntry>}} directly, or a wrapper type carrying per-record keys, or deriving partition keys from each record body.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)