You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ryerraguntla (via GitHub)" <gi...@apache.org> on 2026/10/04 13:08:23 UTC
[PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
ryerraguntla opened a new pull request, #4398:
URL: https://github.com/apache/iggy/pull/4398
## Which issue does this PR address?
Relates to #3546
## Rationale
Kafka admin clients read and change topic retention with DescribeConfigs (32) and the deprecated AlterConfigs (33). Iggy already stores that lifetime as `message_expiry`, and retention applies to sealed segments only. The active segment does not expire. CreateTopics rejects a per-topic config list and tells the client to set `retention.ms` with AlterConfigs, so these two APIs are how a client reads and writes that expiry after the topic exists.
This PR covers the topic-config half only. Unknown keys are `INVALID_CONFIG` (40). Kafka has no `NOT_CONFIGURABLE` code, which is the code the issue text suggested.
## What changed?
With the bridge on, DescribeConfigs (versions 1-4) answers topic resources only. It returns `retention.ms` from Iggy `message_expiry` (`-1` when sealed segments never expire) and a read-only `cleanup.policy=delete`. AlterConfigs (versions 0-2) persists only `retention.ms`, through `update_topic`. A value whose seconds exceed `u32::MAX` is rejected and is not stored. `validate_only` runs the same checks, including the topic lookup, and does not write.
One bad key fails that resource only. Other resources in the same request are still processed. With the bridge off, each resource gets `NOT_CONTROLLER` (41) and nothing is read or written. A missing topic is `UNKNOWN_TOPIC_OR_PARTITION` (3), checked before the config keys. More than 100 distinct topics returns `POLICY_VIOLATION` (44) on every resource in the request.
IncrementalAlterConfigs (44) is not implemented and is not advertised. A client that sends it gets the connection closed. Only the deprecated `Admin.alterConfigs` (33) writes `retention.ms`. CreateTopics still rejects per-topic configs and tells the client to use AlterConfigs. DeleteTopics is not in this PR and is covered in a separate PR. IncrementalAlterConfigs (44) will be implemented in another succeding PRs.
## Local Execution
Passed:
- `cargo build -p iggy-gateway-kafka --tests`
- `cargo clippy -p iggy-gateway-kafka --all-features --all-targets -- -D warnings`
- Unit tests for topic config, bounds, config handlers, version firewall, CreateTopics, and kafka-message-gen
- Kafka 3.9 Java AdminClient probe on the host JDK: `retention.ms=-1`, then alter to `8000`, unknown key rejected, `validate_only` did not write, `describe_batch_50_elapsed_ms=22`. The probe prints that elapsed time and does not assert a latency threshold.
## AI Usage
Fill this in before opening.
1. Which tools? Claude
2. Scope of usage? Implementation and review
3. How did you verify the generated code works correctly? Walk through and manual testing.
4. Can you explain every line of the code if asked? Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4194786643
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
Review Comment:
no authz before an API that deletes data: any admitted client can set `retention.ms=1` and the cleaner drops sealed segments irreversibly (no consumer barrier, the gateway has no OffsetCommit). With SASL on, a read-only principal can do it while DescribeAcls tells it ALTER is denied. README.md:9 warns only about CreateTopics.
With SASL on, gate on `principal.permissions.manage_topics` (auth.rs:512) and answer 29 per resource. At minimum, add the README warning and a CONFIGS.md authz section that names `retention.ms` as destructive.
##########
gateways/kafka/src/protocol/bounds_guard.rs:
##########
@@ -1052,6 +1052,110 @@ pub fn validate_sync_group_shape(version: i16, body: &Bytes, max_frame_size: usi
/// reaches `parse_plain`, and what a future mechanism would carry into a credential exchange.
const MAX_SASL_AUTH_BYTES: usize = 4096;
+/// Mirrors the field order `DescribeConfigsRequest::decode` walks.
+///
+/// # Errors
+///
+/// Returns an error when a declared array or string length cannot fit in the bytes remaining
+/// in the frame, or the body is truncated or malformed in a way that cannot be walked.
+pub fn validate_describe_configs_shape(
Review Comment:
this walker and the AlterConfigs one at :1113 have zero tests, and deleting either keeps everything green. They carry real weight: kafka-protocol 0.18 does `Vec::with_capacity(n)` on decode, so an i32::MAX count from a 4-byte body, before auth, aborts the process on allocation. version_firewall_tests.rs:871,896 claim "rejected by the shape walk" but use 100000, which the crate rejects on its own.
Add cases with i32::MAX, the max compact varint, nested key/config counts, nullable keys, and v3/flexible layouts.
##########
gateways/kafka/docs/CONFIGS.md:
##########
@@ -0,0 +1,131 @@
+# Topic configuration
+
+`DescribeConfigs` (32) and `AlterConfigs` (33) apply to Kafka topic resources only
Review Comment:
writes only work through legacy API 33, and Kafka 4.0 removed `Admin.alterConfigs`. `kafka-configs.sh --alter` on topics, MirrorMaker 2 and Spring send 44 with no fallback. The later claim (:107-109) that a client sending 44 gets its connection closed is false for Java and librdkafka: both fail client-side, since 44 isn't advertised.
Add a client/tool compatibility table and correct that claim.
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
this says the write isn't wrapped in a timeout, but `update_kafka_topic_message_expiry` puts `update_topic` inside `with_request_timeout` (bridge/iggy_bridge/topics.rs:312), so the exact case described here still happens. Timeout or TransientNotCommitted answers 7 while a short retention may still commit and delete sealed segments. The handler can also run 20+15 = 35s, past the 25s shutdown drain (api.rs:201) and Java's 30s `request.timeout.ms`.
Give the write its own budget inside the drain, fix the comment, and on timeout do one bounded re-read to report the real outcome, or document 7 as outcome unknown in CONFIGS.md.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`]. `DescribeConfigs` omits
+/// those names from the response and does not fail the resource.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
+ let Some(keys) = configuration_keys.filter(|keys| !keys.is_empty()) else {
+ return vec![ListedKey::Retention, ListedKey::Cleanup];
+ };
+ keys.iter()
+ .map(|key| match key.as_str() {
+ RETENTION_MS => ListedKey::Retention,
+ RETENTION_MINUTES => ListedKey::Minutes,
+ RETENTION_HOURS => ListedKey::Hours,
+ CLEANUP_POLICY => ListedKey::Cleanup,
+ other => ListedKey::Unknown(other.to_string()),
+ })
+ .collect()
+}
+
+/// The expiry to store, or `None` when the resource names no retention key.
+///
+/// `retention.ms`, `retention.minutes`, and `retention.hours` are one duration.
+/// Minutes and hours convert to milliseconds. Two names that convert to
+/// different millisecond values fail the resource and select nothing. One
+/// expiry is returned when they agree. [`PlannedRetention::synonyms`] is empty
+/// when the resource used only `retention.ms`.
+///
+/// One invalid or repeated key fails the whole resource. Nothing is selected
+/// for storage in that case. `update_topic` stores the millisecond expiry, never
+/// the minute or hour count.
+///
+/// # Errors
+///
+/// Returns the first [`ConfigFault`] in request order.
+pub fn plan_retention_update<'a>(
+ configs: impl IntoIterator<Item = (&'a str, Option<&'a str>)>,
+) -> Result<Option<PlannedRetention>, ConfigFault> {
+ let mut seen = HashSet::new();
+ let mut expiry = None;
+ let mut synonyms = RetentionSynonyms::default();
+ for (name, value) in configs {
+ if !seen.insert(name) {
+ return Err(ConfigFault::Repeated(name.to_string()));
+ }
+ let unit_ms = retention_scale(name, &mut synonyms)?;
+ let Some(value) = value else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ let parsed = parse_scaled_retention(value, unit_ms)?;
+ if let Some(existing) = expiry
+ && existing != parsed
+ {
+ return Err(ConfigFault::ConflictingRetention);
+ }
+ expiry = Some(parsed);
+ }
+ Ok(expiry.map(|expiry| PlannedRetention { expiry, synonyms }))
+}
+
+fn retention_scale(name: &str, synonyms: &mut RetentionSynonyms) -> Result<u64, ConfigFault> {
Review Comment:
`retention.minutes` / `retention.hours` aren't Kafka topic configs. Only broker `log.retention.*` exists, and a broker answers 40 "Unknown topic config name" for them. Describe then reports them as DYNAMIC_TOPIC_CONFIG synonyms from `RetentionSynonymMemory` (:112), which is per-instance process memory that is lost on restart and stale after any external change.
Reject both with 40 and drop the memory. If synonyms are wanted, report `retention.ms` itself, Kafka style.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
Review Comment:
none of the guards are pinned. These mutations stay green: dropping this 100-topic cap, the deadline-before-write check (alter :290), the tagged-field reject and record-synonyms. The cap also sits after the bridge-off return (describe_configs.rs:80, alter_configs.rs:74), so no serverless test can reach it.
Move the cap check before the bridge check and add 100/101 boundary tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4179272512
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(state, bridge, resource, req.validate_only, deadline).await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ if Instant::now() >= deadline {
+ tracing::warn!(
+ resource = name,
+ "AlterConfigs deadline passed before the topic lookup"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ match tokio::time::timeout_at(deadline, bridge.get_kafka_topic(name)).await {
+ Ok(Ok(Some(_topic))) => {}
+ Ok(Ok(None)) => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ Ok(Err(error)) => {
+ let (code, message) = bridge_failure(&error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ Err(_elapsed) => {
+ tracing::warn!(
+ resource = name,
+ "AlterConfigs: this resource's bridge work exceeded the request deadline; \
+ answering retriable instead of blocking further"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // A resource that names no retention key has nothing to store. Omitting the key
+ // does not clear a previously set expiry or the remembered synonym names.
+ let Some(planned) = planned else {
Review Comment:
warning: An empty `configs` list makes `AlterConfigs` answer `ERROR_NONE` and write nothing, but Kafka's legacy `AlterConfigs` replaces the full config set, so a client that asks for a reset sees success. Reject the empty list with `INVALID_CONFIG` (40), and state the replace-to-patch deviation in `docs/CONFIGS.md`.
##########
gateways/kafka/tests/configs_java_client_tests.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Kafka 3.9 `AdminClient` describe/alter against the gateway, on the host JDK.
+//!
+//! The docker `--network host` suite cannot reach a loopback gateway from macOS.
+//! This test publishes nothing into a container: it binds `127.0.0.1` and runs
+//! `java` on the host. Jars come from Maven Central when they are not cached.
+
+use std::io::Read;
+use std::net::SocketAddr;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Output, Stdio};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use tokio::sync::broadcast;
+
+use iggy_gateway_kafka::bridge::IggyBridge;
+use iggy_gateway_kafka::server::bind_listener;
+use iggy_gateway_kafka::{GatewayConfig, KafkaGateway};
+
+#[path = "common/iggy_server.rs"]
+mod iggy_server;
+
+use iggy_server::TestServer;
+
+const TOPIC: &str = "orders";
+
+/// Bounds a child the way `kafka_client_e2e_tests.rs` bounds a client run, without GNU `timeout`
+/// (this suite runs on the host JDK, including macOS). A hung `java` or `curl` used to block the
+/// process until the runner itself was killed.
+const CHILD_LIMIT: Duration = Duration::from_secs(120);
+
+/// Reports why the suite cannot run, and whether that is fatal.
+///
+/// Returns `true` when the caller should skip. `JAVA_E2E_REQUIRED=1` makes it panic instead, so a
+/// CI job that means to run this fails loudly rather than reporting a pass over zero assertions -
+/// same convention as `kafka_client_e2e_tests.rs`'s own `skip`/`KAFKA_E2E_REQUIRED`.
+fn skip(reason: &str) -> bool {
+ assert!(
+ std::env::var("JAVA_E2E_REQUIRED").as_deref() != Ok("1"),
+ "JAVA_E2E_REQUIRED=1 but the suite cannot run: {reason}"
+ );
+ eprintln!("skipping Java admin-client end-to-end test: {reason}");
+ true
+}
+
+fn java_missing() -> bool {
+ let available = Command::new("java")
+ .arg("-version")
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .status()
+ .is_ok_and(|status| status.success());
+ if available {
+ false
+ } else {
+ skip("java is unavailable")
+ }
+}
+
+fn output_bounded(command: &mut Command, what: &str) -> Output {
+ let mut child = command
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .unwrap_or_else(|error| panic!("spawn {what}: {error}"));
+ let mut stdout = child.stdout.take().expect("stdout");
+ let mut stderr = child.stderr.take().expect("stderr");
+ let stdout_thread = std::thread::spawn(move || {
+ let mut buf = Vec::new();
+ stdout.read_to_end(&mut buf).map(|_read| buf)
+ });
+ let stderr_thread = std::thread::spawn(move || {
+ let mut buf = Vec::new();
+ stderr.read_to_end(&mut buf).map(|_read| buf)
+ });
+ let deadline = Instant::now() + CHILD_LIMIT;
+ let status = loop {
+ match child.try_wait() {
+ Ok(Some(status)) => break status,
+ Ok(None) if Instant::now() >= deadline => {
+ let _ = child.kill();
+ let _ = child.wait();
+ panic!("{what} exceeded {CHILD_LIMIT:?}");
+ }
+ Ok(None) => std::thread::sleep(Duration::from_millis(50)),
+ Err(error) => panic!("poll {what}: {error}"),
+ }
+ };
+ let stdout = stdout_thread
+ .join()
+ .unwrap_or_else(|_| panic!("{what} stdout thread"))
+ .unwrap_or_else(|error| panic!("read {what} stdout: {error}"));
+ let stderr = stderr_thread
+ .join()
+ .unwrap_or_else(|_| panic!("{what} stderr thread"))
+ .unwrap_or_else(|error| panic!("read {what} stderr: {error}"));
+ Output {
+ status,
+ stdout,
+ stderr,
+ }
+}
+
+const JARS: &[(&str, &str)] = &[
+ (
+ "kafka-clients-3.9.0.jar",
+ "https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.9.0/kafka-clients-3.9.0.jar",
+ ),
+ (
+ "slf4j-api-1.7.36.jar",
+ "https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar",
+ ),
+ (
+ "slf4j-simple-1.7.36.jar",
+ "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/1.7.36/slf4j-simple-1.7.36.jar",
+ ),
+ (
+ "lz4-java-1.8.0.jar",
+ "https://repo1.maven.org/maven2/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar",
+ ),
+ (
+ "snappy-java-1.1.10.5.jar",
+ "https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.10.5/snappy-java-1.1.10.5.jar",
+ ),
+ (
+ "zstd-jni-1.5.6-4.jar",
+ "https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.5.6-4/zstd-jni-1.5.6-4.jar",
+ ),
+];
+
+fn jar_dir() -> PathBuf {
+ PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target/kafka-3.9-admin-jars")
+}
+
+fn ensure_jars(dir: &Path) {
+ std::fs::create_dir_all(dir).expect("create jar cache");
+ for (name, url) in JARS {
+ let path = dir.join(name);
+ if path.exists() {
+ continue;
+ }
+ let output = output_bounded(
+ Command::new("curl")
+ .args([
+ "--fail",
+ "--silent",
+ "--show-error",
+ "--location",
+ "--max-time",
+ "60",
+ "-o",
+ ])
+ .arg(&path)
+ .arg(url),
+ name,
+ );
+ if !output.status.success() {
+ let _ = std::fs::remove_file(&path);
+ }
+ assert!(
Review Comment:
nit: `ensure_jars()` panics when `curl` fails, even with `JAVA_E2E_REQUIRED` unset, so an offline machine with a JDK fails the suite instead of skipping. Make it report the failure so the caller can call `skip()`.
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,681 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::TopicDetails;
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::IggyBridge;
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, CONFIG_TYPE_UNKNOWN, IggyTopicKey, ListedKey,
+ MILLIS_PER_HOUR, MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC,
+ RETENTION_HOURS, RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS,
+ RetentionMs, RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names,
+ listed_keys, message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms,
+ static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ DescribeConfigsResult::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("DescribeConfigs"));
+ tracing::warn!("DescribeConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_POLICY_VIOLATION,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // answering it from a cached result that may have been filtered by a different
+ // `configuration_keys` list - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut results = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ results.push(resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "DescribeConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ results.push(resource_error(
+ resource,
+ ERROR_REQUEST_TIMED_OUT,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+
+ let result = describe_one(
+ state,
+ bridge,
+ resource,
+ req.include_synonyms,
+ req.include_documentation,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ results.push(result);
+ }
+ results
+}
+
+async fn describe_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ resource: &DescribeConfigsResource,
+ include_synonyms: bool,
+ include_documentation: bool,
+ deadline: Instant,
+) -> DescribeConfigsResult {
+ if !resource.unknown_tagged_fields.is_empty() {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ Vec::new(),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ Vec::new(),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ Vec::new(),
+ );
+ }
+
+ if Instant::now() >= deadline {
+ tracing::warn!(
+ resource = name,
+ "DescribeConfigs deadline passed before the topic lookup"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ let topic = match tokio::time::timeout_at(deadline, bridge.get_kafka_topic(name)).await {
Review Comment:
warning: The resource loop pays one `get_kafka_topic()` round trip per resource, so a request naming 100 topics makes 100 sequential calls on the shared lockstep client, and each one blocks Produce and Metadata. Group resources by resolved stream and call `get_topics` once per stream, as `get_kafka_topics()` does.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,725 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+/// Kafka `ConfigType.UNKNOWN`.
+pub const CONFIG_TYPE_UNKNOWN: i8 = 0;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct Kafka topic names one `DescribeConfigs` or `AlterConfigs` request may send to Iggy.
+///
+/// Same ceiling as `CreateTopics`: each distinct name is a lockstep call on the shared client.
+/// Duplicate names in one request count once for this cap, but a repeated name is itself
+/// rejected outright - see [`find_duplicate_names`]. A larger batch is `POLICY_VIOLATION` on
+/// every resource. The decode-time element ceiling is a separate, larger limit and is not this
+/// cap.
+pub const MAX_CONFIG_TOPICS: usize = 100;
Review Comment:
simplification: `MAX_CONFIG_TOPICS` repeats the 100-topic cap that `create_topics.rs` defines, and `topic_cap_message()` repeats its message. Declare the cap and message once and use them in both modules.
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,681 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::TopicDetails;
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::IggyBridge;
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, CONFIG_TYPE_UNKNOWN, IggyTopicKey, ListedKey,
+ MILLIS_PER_HOUR, MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC,
+ RETENTION_HOURS, RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS,
+ RetentionMs, RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names,
+ listed_keys, message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms,
+ static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ DescribeConfigsResult::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("DescribeConfigs"));
+ tracing::warn!("DescribeConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_POLICY_VIOLATION,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // answering it from a cached result that may have been filtered by a different
+ // `configuration_keys` list - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut results = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ results.push(resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "DescribeConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ results.push(resource_error(
+ resource,
+ ERROR_REQUEST_TIMED_OUT,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+
+ let result = describe_one(
+ state,
+ bridge,
+ resource,
+ req.include_synonyms,
+ req.include_documentation,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ results.push(result);
+ }
+ results
+}
+
+async fn describe_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ resource: &DescribeConfigsResource,
+ include_synonyms: bool,
+ include_documentation: bool,
+ deadline: Instant,
+) -> DescribeConfigsResult {
+ if !resource.unknown_tagged_fields.is_empty() {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ Vec::new(),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ Vec::new(),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ Vec::new(),
+ );
+ }
+
+ if Instant::now() >= deadline {
+ tracing::warn!(
+ resource = name,
+ "DescribeConfigs deadline passed before the topic lookup"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ let topic = match tokio::time::timeout_at(deadline, bridge.get_kafka_topic(name)).await {
+ Ok(Ok(Some(topic))) => topic,
+ Ok(Ok(None)) => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None, Vec::new());
+ }
+ Ok(Err(error)) => {
+ let (code, message) = bridge_failure(&error, "DescribeConfigs", "reading");
+ return resource_error(resource, code, message, Vec::new());
+ }
+ Err(_elapsed) => {
+ tracing::warn!(
+ resource = name,
+ "DescribeConfigs: this resource's bridge work exceeded the request deadline; \
+ answering retriable instead of blocking further"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ };
+
+ let (stream, iggy_topic) = bridge.topic_identity(name);
+ let remembered = state.remembered_retention_synonyms(&IggyTopicKey::new(stream, iggy_topic));
+ let (configs, unknown_key, unrepresentable) = described_entries(
+ &topic,
+ resource.configuration_keys.as_deref(),
+ include_synonyms,
+ include_documentation,
+ remembered,
+ );
+
+ if let Some(unknown_key) = unknown_key {
Review Comment:
warning: An unmodeled requested key fails the whole resource with `INVALID_CONFIG` (40), and clients discard configs on error, so such a request loses `retention.ms` too. Answer `ERROR_NONE` and omit the unknown key, as a real broker does.
##########
gateways/kafka/tests/version_firewall_tests.rs:
##########
@@ -101,7 +103,7 @@ fn is_supported_version_matches_scope_table() {
///
/// Relies on `SUPPORTED_RANGES` (src) and `SCOPED_API_KEYS` (test) sharing declaration order
/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`, `FindCoordinator`,
-/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) - `supported_ranges_table_has_twelve_entries` plus
+/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) - `supported_ranges_table_has_fourteen_entries` plus
Review Comment:
nit: The comment says `SUPPORTED_RANGES` and `SCOPED_API_KEYS` share declaration order, but `api_versions.rs` sorts the wire rows by `api_key` and the scope table is the ascending one. Rewrite it to say that the test relies on that sort.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,725 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+/// Kafka `ConfigType.UNKNOWN`.
+pub const CONFIG_TYPE_UNKNOWN: i8 = 0;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct Kafka topic names one `DescribeConfigs` or `AlterConfigs` request may send to Iggy.
+///
+/// Same ceiling as `CreateTopics`: each distinct name is a lockstep call on the shared client.
+/// Duplicate names in one request count once for this cap, but a repeated name is itself
+/// rejected outright - see [`find_duplicate_names`]. A larger batch is `POLICY_VIOLATION` on
+/// every resource. The decode-time element ceiling is a separate, larger limit and is not this
+/// cap.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`] and stays in the list.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
+ let Some(keys) = configuration_keys.filter(|keys| !keys.is_empty()) else {
+ return vec![ListedKey::Retention, ListedKey::Cleanup];
+ };
+ keys.iter()
+ .map(|key| match key.as_str() {
+ RETENTION_MS => ListedKey::Retention,
+ RETENTION_MINUTES => ListedKey::Minutes,
+ RETENTION_HOURS => ListedKey::Hours,
+ CLEANUP_POLICY => ListedKey::Cleanup,
+ other => ListedKey::Unknown(other.to_string()),
+ })
+ .collect()
+}
+
+/// The expiry to store, or `None` when the resource names no retention key.
+///
+/// `retention.ms`, `retention.minutes`, and `retention.hours` are one duration.
+/// Minutes and hours convert to milliseconds. Two names that convert to
+/// different millisecond values fail the resource and select nothing. One
+/// expiry is returned when they agree. [`PlannedRetention::synonyms`] is empty
+/// when the resource used only `retention.ms`.
+///
+/// One invalid or repeated key fails the whole resource. Nothing is selected
+/// for storage in that case. `update_topic` stores the millisecond expiry, never
+/// the minute or hour count.
+///
+/// # Errors
+///
+/// Returns the first [`ConfigFault`] in request order.
+pub fn plan_retention_update<'a>(
+ configs: impl IntoIterator<Item = (&'a str, Option<&'a str>)>,
+) -> Result<Option<PlannedRetention>, ConfigFault> {
+ let mut seen = HashSet::new();
+ let mut expiry = None;
+ let mut synonyms = RetentionSynonyms::default();
+ for (name, value) in configs {
+ if !seen.insert(name) {
+ return Err(ConfigFault::Repeated(name.to_string()));
+ }
+ let unit_ms = retention_scale(name, &mut synonyms)?;
+ let Some(value) = value else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ let parsed = if unit_ms == 1 {
Review Comment:
simplification: The `unit_ms == 1` branch calls `parse_retention_ms(value)`, which is exactly `parse_scaled_retention(value, 1)`. Delete the branch and call `parse_scaled_retention(value, unit_ms)` for every unit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4179487053
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,681 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::TopicDetails;
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::IggyBridge;
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, CONFIG_TYPE_UNKNOWN, IggyTopicKey, ListedKey,
+ MILLIS_PER_HOUR, MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC,
+ RETENTION_HOURS, RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS,
+ RetentionMs, RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names,
+ listed_keys, message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms,
+ static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ DescribeConfigsResult::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("DescribeConfigs"));
+ tracing::warn!("DescribeConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_POLICY_VIOLATION,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // answering it from a cached result that may have been filtered by a different
+ // `configuration_keys` list - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut results = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ results.push(resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "DescribeConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ results.push(resource_error(
+ resource,
+ ERROR_REQUEST_TIMED_OUT,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+
+ let result = describe_one(
+ state,
+ bridge,
+ resource,
+ req.include_synonyms,
+ req.include_documentation,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ results.push(result);
+ }
+ results
+}
+
+async fn describe_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ resource: &DescribeConfigsResource,
+ include_synonyms: bool,
+ include_documentation: bool,
+ deadline: Instant,
+) -> DescribeConfigsResult {
+ if !resource.unknown_tagged_fields.is_empty() {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ Vec::new(),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ Vec::new(),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ Vec::new(),
+ );
+ }
+
+ if Instant::now() >= deadline {
+ tracing::warn!(
+ resource = name,
+ "DescribeConfigs deadline passed before the topic lookup"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ let topic = match tokio::time::timeout_at(deadline, bridge.get_kafka_topic(name)).await {
+ Ok(Ok(Some(topic))) => topic,
+ Ok(Ok(None)) => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None, Vec::new());
+ }
+ Ok(Err(error)) => {
+ let (code, message) = bridge_failure(&error, "DescribeConfigs", "reading");
+ return resource_error(resource, code, message, Vec::new());
+ }
+ Err(_elapsed) => {
+ tracing::warn!(
+ resource = name,
+ "DescribeConfigs: this resource's bridge work exceeded the request deadline; \
+ answering retriable instead of blocking further"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ };
+
+ let (stream, iggy_topic) = bridge.topic_identity(name);
+ let remembered = state.remembered_retention_synonyms(&IggyTopicKey::new(stream, iggy_topic));
+ let (configs, unknown_key, unrepresentable) = described_entries(
+ &topic,
+ resource.configuration_keys.as_deref(),
+ include_synonyms,
+ include_documentation,
+ remembered,
+ );
+
+ if let Some(unknown_key) = unknown_key {
Review Comment:
fixed
##########
gateways/kafka/tests/version_firewall_tests.rs:
##########
@@ -101,7 +103,7 @@ fn is_supported_version_matches_scope_table() {
///
/// Relies on `SUPPORTED_RANGES` (src) and `SCOPED_API_KEYS` (test) sharing declaration order
/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`, `FindCoordinator`,
-/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) - `supported_ranges_table_has_twelve_entries` plus
+/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) - `supported_ranges_table_has_fourteen_entries` plus
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205906597
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4179464032
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,681 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::TopicDetails;
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::IggyBridge;
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, CONFIG_TYPE_UNKNOWN, IggyTopicKey, ListedKey,
+ MILLIS_PER_HOUR, MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC,
+ RETENTION_HOURS, RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS,
+ RetentionMs, RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names,
+ listed_keys, message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms,
+ static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ DescribeConfigsResult::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("DescribeConfigs"));
+ tracing::warn!("DescribeConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_POLICY_VIOLATION,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // answering it from a cached result that may have been filtered by a different
+ // `configuration_keys` list - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut results = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ results.push(resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "DescribeConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ results.push(resource_error(
+ resource,
+ ERROR_REQUEST_TIMED_OUT,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+
+ let result = describe_one(
+ state,
+ bridge,
+ resource,
+ req.include_synonyms,
+ req.include_documentation,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ results.push(result);
+ }
+ results
+}
+
+async fn describe_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ resource: &DescribeConfigsResource,
+ include_synonyms: bool,
+ include_documentation: bool,
+ deadline: Instant,
+) -> DescribeConfigsResult {
+ if !resource.unknown_tagged_fields.is_empty() {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ Vec::new(),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ Vec::new(),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ Vec::new(),
+ );
+ }
+
+ if Instant::now() >= deadline {
+ tracing::warn!(
+ resource = name,
+ "DescribeConfigs deadline passed before the topic lookup"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ let topic = match tokio::time::timeout_at(deadline, bridge.get_kafka_topic(name)).await {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205913364
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`]. `DescribeConfigs` omits
+/// those names from the response and does not fail the resource.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
Review Comment:
fixed
##########
gateways/kafka/README.md:
##########
@@ -1,11 +1,12 @@
# Kafka gateway (`iggy-gateway-kafka`)
-Foundation layer for [apache/iggy#3421](https://github.com/apache/iggy/issues/3421): a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata and CreateTopics use Iggy. InitProducerId and consumer group coordination work with or without one.
+Foundation layer for [apache/iggy#3421](https://github.com/apache/iggy/issues/3421): a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata, CreateTopics, DescribeConfigs and AlterConfigs use Iggy. InitProducerId and consumer group coordination work with or without one.
-> **Stub warning:** with `IGGY_KAFKA_BRIDGE_ENABLED=true`, Produce, Fetch, ListOffsets, Metadata and
-> CreateTopics use Iggy. With the bridge off (the default), those five are stubs: Produce, Fetch and
-> ListOffsets answer retriable `NOT_LEADER_OR_FOLLOWER` (6), CreateTopics answers `NOT_CONTROLLER`
-> (41), and Metadata reports every requested topic unknown. **CreateTopics runs as the bridge's own
+> **Stub warning:** with `IGGY_KAFKA_BRIDGE_ENABLED=true`, Produce, Fetch, ListOffsets, Metadata,
+> CreateTopics, DescribeConfigs and AlterConfigs use Iggy. With the bridge off (the default), those
+> seven are stubs: Produce, Fetch and ListOffsets answer retriable `NOT_LEADER_OR_FOLLOWER` (6),
+> CreateTopics, DescribeConfigs and AlterConfigs answer `NOT_CONTROLLER` (41), and Metadata reports
+> every requested topic unknown. **CreateTopics runs as the bridge's own
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4194392122
##########
gateways/kafka/README.md:
##########
@@ -1,11 +1,12 @@
# Kafka gateway (`iggy-gateway-kafka`)
-Foundation layer for [apache/iggy#3421](https://github.com/apache/iggy/issues/3421): a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata and CreateTopics use Iggy. InitProducerId and consumer group coordination work with or without one.
+Foundation layer for [apache/iggy#3421](https://github.com/apache/iggy/issues/3421): a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata, CreateTopics, DescribeConfigs and AlterConfigs use Iggy. InitProducerId and consumer group coordination work with or without one.
-> **Stub warning:** with `IGGY_KAFKA_BRIDGE_ENABLED=true`, Produce, Fetch, ListOffsets, Metadata and
-> CreateTopics use Iggy. With the bridge off (the default), those five are stubs: Produce, Fetch and
-> ListOffsets answer retriable `NOT_LEADER_OR_FOLLOWER` (6), CreateTopics answers `NOT_CONTROLLER`
-> (41), and Metadata reports every requested topic unknown. **CreateTopics runs as the bridge's own
+> **Stub warning:** with `IGGY_KAFKA_BRIDGE_ENABLED=true`, Produce, Fetch, ListOffsets, Metadata,
+> CreateTopics, DescribeConfigs and AlterConfigs use Iggy. With the bridge off (the default), those
+> seven are stubs: Produce, Fetch and ListOffsets answer retriable `NOT_LEADER_OR_FOLLOWER` (6),
+> CreateTopics, DescribeConfigs and AlterConfigs answer `NOT_CONTROLLER` (41), and Metadata reports
+> every requested topic unknown. **CreateTopics runs as the bridge's own
Review Comment:
With the bridge on, any client that reaches this port can now also shorten retention through AlterConfigs, and Iggy then deletes the old sealed segments.
Please add AlterConfigs to this warning and say that it can delete data.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`]. `DescribeConfigs` omits
+/// those names from the response and does not fail the resource.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
Review Comment:
`listed_keys` keeps repeated keys. A 56 KiB v3+ request that repeats `retention.ms` with documentation on gets 0.6 to 0.9 MB back, 2 to 3 times what `bounds_guard` allows for.
A Kafka broker returns each name once. Same here, one entry per known key per resource.
##########
gateways/kafka/tests/configs_java_client_tests.rs:
##########
@@ -0,0 +1,310 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Kafka 3.9 `AdminClient` describe/alter against the gateway, on the host JDK.
+//!
+//! The docker `--network host` suite cannot reach a loopback gateway from macOS.
+//! This test publishes nothing into a container: it binds `127.0.0.1` and runs
+//! `java` on the host. Jars come from Maven Central when they are not cached.
+
+use std::io::Read;
+use std::net::SocketAddr;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Output, Stdio};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use tokio::sync::broadcast;
+
+use iggy_gateway_kafka::bridge::IggyBridge;
+use iggy_gateway_kafka::server::bind_listener;
+use iggy_gateway_kafka::{GatewayConfig, KafkaGateway};
+
+#[path = "common/iggy_server.rs"]
+mod iggy_server;
+
+use iggy_server::TestServer;
+
+const TOPIC: &str = "orders";
+
+/// Bounds a child the way `kafka_client_e2e_tests.rs` bounds a client run, without GNU `timeout`
+/// (this suite runs on the host JDK, including macOS). A hung `java` or `curl` used to block the
+/// process until the runner itself was killed.
+const CHILD_LIMIT: Duration = Duration::from_secs(120);
+
+/// Reports why the suite cannot run, and whether that is fatal.
+///
+/// Returns `true` when the caller should skip. `JAVA_E2E_REQUIRED=1` makes it panic instead, so a
+/// CI job that means to run this fails loudly rather than reporting a pass over zero assertions -
+/// same convention as `kafka_client_e2e_tests.rs`'s own `skip`/`KAFKA_E2E_REQUIRED`.
+fn skip(reason: &str) -> bool {
+ assert!(
+ std::env::var("JAVA_E2E_REQUIRED").as_deref() != Ok("1"),
+ "JAVA_E2E_REQUIRED=1 but the suite cannot run: {reason}"
+ );
+ eprintln!("skipping Java admin-client end-to-end test: {reason}");
+ true
+}
+
+fn java_missing() -> bool {
+ let available = Command::new("java")
+ .arg("-version")
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .status()
+ .is_ok_and(|status| status.success());
+ if available {
+ false
+ } else {
+ skip("java is unavailable")
+ }
+}
+
+fn output_bounded(command: &mut Command, what: &str) -> Output {
+ try_output_bounded(command, what).unwrap_or_else(|error| panic!("{error}"))
+}
+
+fn try_output_bounded(command: &mut Command, what: &str) -> Result<Output, String> {
+ let mut child = command
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .map_err(|error| format!("spawn {what}: {error}"))?;
+ let mut stdout = child
+ .stdout
+ .take()
+ .ok_or_else(|| format!("{what} stdout"))?;
+ let mut stderr = child
+ .stderr
+ .take()
+ .ok_or_else(|| format!("{what} stderr"))?;
+ let stdout_thread = std::thread::spawn(move || {
+ let mut buf = Vec::new();
+ stdout.read_to_end(&mut buf).map(|_read| buf)
+ });
+ let stderr_thread = std::thread::spawn(move || {
+ let mut buf = Vec::new();
+ stderr.read_to_end(&mut buf).map(|_read| buf)
+ });
+ let deadline = Instant::now() + CHILD_LIMIT;
+ let status = loop {
+ match child.try_wait() {
+ Ok(Some(status)) => break status,
+ Ok(None) if Instant::now() >= deadline => {
+ let _ = child.kill();
+ let _ = child.wait();
+ return Err(format!("{what} exceeded {CHILD_LIMIT:?}"));
+ }
+ Ok(None) => std::thread::sleep(Duration::from_millis(50)),
+ Err(error) => return Err(format!("poll {what}: {error}")),
+ }
+ };
+ let stdout = stdout_thread
+ .join()
+ .map_err(|_| format!("{what} stdout thread"))?
+ .map_err(|error| format!("read {what} stdout: {error}"))?;
+ let stderr = stderr_thread
+ .join()
+ .map_err(|_| format!("{what} stderr thread"))?
+ .map_err(|error| format!("read {what} stderr: {error}"))?;
+ Ok(Output {
+ status,
+ stdout,
+ stderr,
+ })
+}
+
+const JARS: &[(&str, &str)] = &[
+ (
+ "kafka-clients-3.9.0.jar",
+ "https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.9.0/kafka-clients-3.9.0.jar",
+ ),
+ (
+ "slf4j-api-1.7.36.jar",
+ "https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar",
+ ),
+ (
+ "slf4j-simple-1.7.36.jar",
+ "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/1.7.36/slf4j-simple-1.7.36.jar",
+ ),
+ (
+ "lz4-java-1.8.0.jar",
+ "https://repo1.maven.org/maven2/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar",
+ ),
+ (
+ "snappy-java-1.1.10.5.jar",
+ "https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.10.5/snappy-java-1.1.10.5.jar",
+ ),
+ (
+ "zstd-jni-1.5.6-4.jar",
+ "https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.5.6-4/zstd-jni-1.5.6-4.jar",
+ ),
+];
+
+fn jar_dir() -> PathBuf {
+ PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target/kafka-3.9-admin-jars")
+}
+
+fn ensure_jars(dir: &Path) -> Result<(), String> {
+ std::fs::create_dir_all(dir).map_err(|error| format!("create jar cache: {error}"))?;
+ for (name, url) in JARS {
+ let path = dir.join(name);
+ if path.exists() {
+ continue;
+ }
+ let output = match try_output_bounded(
+ Command::new("curl")
+ .args([
+ "--fail",
+ "--silent",
+ "--show-error",
+ "--location",
+ "--max-time",
+ "60",
+ "-o",
+ ])
+ .arg(&path)
+ .arg(url),
+ name,
+ ) {
+ Ok(output) => output,
+ Err(error) => {
+ let _ = std::fs::remove_file(&path);
+ return Err(error);
+ }
+ };
+ if !output.status.success() {
+ let _ = std::fs::remove_file(&path);
+ return Err(format!(
+ "download {url} failed: {}{}",
+ output.status,
+ String::from_utf8_lossy(&output.stderr)
+ ));
+ }
+ }
+ Ok(())
+}
+
+fn classpath(jars: &Path, classes: &Path) -> String {
+ let mut parts = vec![classes.display().to_string()];
+ for (name, _) in JARS {
+ parts.push(jars.join(name).display().to_string());
+ }
+ parts.join(":")
+}
+
+async fn spawn_gateway(server: &TestServer) -> SocketAddr {
+ let bridge = IggyBridge::connect(server.test_config())
+ .await
+ .expect("bridge connects");
+ bridge
+ .ensure_stream_and_topic(TOPIC, 1)
+ .await
+ .expect("create topic");
+ let listener = bind_listener("127.0.0.1:0").expect("bind");
+ let addr = listener.local_addr().expect("local addr");
+ let gateway = KafkaGateway::new(GatewayConfig {
+ bind_addr: addr.to_string(),
+ ..GatewayConfig::default()
+ })
+ .with_bridge(Some(Arc::new(bridge)));
+ let (shutdown, receiver) = broadcast::channel::<()>(1);
+ tokio::spawn(async move {
+ let _ = gateway.run(listener, receiver).await;
+ });
+ std::mem::forget(shutdown);
+ addr
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn kafka_39_admin_client_describes_and_alters_topic_configs() {
+ if java_missing() {
+ return;
+ }
+
+ let jars = jar_dir();
+ if let Err(reason) = ensure_jars(&jars)
Review Comment:
CI never sets `JAVA_E2E_REQUIRED=1`. If `java` is missing or a jar download fails, this test skips and still passes.
Export it and fetch the jars next to the `KAFKA_E2E_REQUIRED=1` export in `.github/actions/rust/pre-merge/action.yml`.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
Review Comment:
Some of these checks can never fail: `unit_ms == 0` (:262, :287), `u64::try_from` right after the sign checks (:265, :285), and `checked_mul(1_000)` after the seconds cap (:297).
`count.unsigned_abs()` with `checked_mul(unit_ms)` plus the seconds cap is enough.
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ DescribeConfigsResult::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("DescribeConfigs"));
+ tracing::warn!("DescribeConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_POLICY_VIOLATION,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // answering it from a cached result that may have been filtered by a different
+ // `configuration_keys` list - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut results = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ results.push(resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "DescribeConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ results.push(resource_error(
+ resource,
+ ERROR_REQUEST_TIMED_OUT,
+ None,
+ Vec::new(),
+ ));
+ continue;
+ }
+
+ let result = describe_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.include_synonyms,
+ req.include_documentation,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ results.push(result);
+ }
+ results
+}
+
+async fn describe_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &DescribeConfigsResource,
+ include_synonyms: bool,
+ include_documentation: bool,
+ deadline: Instant,
+) -> DescribeConfigsResult {
+ if !resource.unknown_tagged_fields.is_empty() {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ Vec::new(),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ Vec::new(),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ Vec::new(),
+ );
+ }
+
+ let snapshot = match cache
+ .lookup(bridge, name, deadline, "DescribeConfigs")
+ .await
+ {
+ TopicLoad::Found(snapshot) => snapshot,
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None, Vec::new());
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "DescribeConfigs", "reading");
+ return resource_error(resource, code, message, Vec::new());
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None, Vec::new());
+ }
+ };
+
+ let (stream, iggy_topic) = bridge.topic_identity(name);
+ let remembered = state.remembered_retention_synonyms(&IggyTopicKey::new(stream, iggy_topic));
+ let (configs, unrepresentable) = described_entries(
+ snapshot.message_expiry,
+ &snapshot.options,
+ resource.configuration_keys.as_deref(),
+ include_synonyms,
+ include_documentation,
+ remembered,
+ );
+
+ if unrepresentable {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "stored message expiry is not a whole number of milliseconds",
+ )),
+ configs,
+ );
+ }
+ resource_error(resource, ERROR_NONE, None, configs)
+}
+
+#[must_use]
+fn described_entries(
+ message_expiry: IggyExpiry,
+ options: &ResourceOptions,
+ configuration_keys: Option<&[StrBytes]>,
+ include_synonyms: bool,
+ include_documentation: bool,
+ remembered: RetentionSynonyms,
+) -> (Vec<DescribeConfigsResourceResult>, bool) {
+ let explicit = message_expiry_is_explicit(options);
+ let retention = retention_ms(message_expiry, explicit);
+ let keys = listed_keys(configuration_keys);
+ let mut unrepresentable = false;
+ let mut configs = Vec::with_capacity(keys.len());
+ for key in keys {
+ match key {
+ ListedKey::Retention => {
+ let (value, source) = retention_parts(&retention, &mut unrepresentable);
+ configs.push(config_entry(
+ RETENTION_MS,
+ value,
+ false,
+ source,
+ CONFIG_TYPE_LONG,
+ synonyms(include_synonyms, remembered, value, source),
+ include_documentation.then_some(RETENTION_DOC),
+ ));
+ }
+ ListedKey::Minutes => configs.push(derived_retention_entry(
+ RETENTION_MINUTES,
+ MILLIS_PER_MINUTE,
+ &retention,
+ include_documentation.then_some(RETENTION_MINUTES_DOC),
+ &mut unrepresentable,
+ )),
+ ListedKey::Hours => configs.push(derived_retention_entry(
+ RETENTION_HOURS,
+ MILLIS_PER_HOUR,
+ &retention,
+ include_documentation.then_some(RETENTION_HOURS_DOC),
+ &mut unrepresentable,
+ )),
+ ListedKey::Cleanup => configs.push(config_entry(
+ CLEANUP_POLICY,
+ Some(CLEANUP_POLICY_VALUE),
+ true,
+ CONFIG_SOURCE_DEFAULT,
+ CONFIG_TYPE_STRING,
+ Vec::new(),
+ include_documentation.then_some(CLEANUP_DOC),
+ )),
+ // A resource error makes clients drop every returned config, including
+ // retention.ms. An unmodeled name is omitted and the resource stays successful.
+ ListedKey::Unknown(_) => {}
+ }
+ }
+ (configs, unrepresentable)
+}
+
+const fn retention_parts<'a>(
Review Comment:
`retention_parts` and `derived_retention_entry` pass a `&mut bool` around to record one fact per resource.
Compute `(millis, source)` and that flag once before the loop instead.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
Review Comment:
`static_text` only forwards to `StrBytes::from_static_str`, which the rest of the crate calls directly.
Call it directly at the eight call sites and drop the wrapper.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
Review Comment:
This test-only wrapper holds the only comment that explains the `-1` rule.
Move that doc onto `parse_scaled_retention`, make it `pub(crate)`, and drop the wrapper.
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
The comment says that nothing wraps this write in a timeout, but `with_request_timeout` (15 s, topics.rs:312) does. We can still answer REQUEST_TIMED_OUT while the write commits later, and a late batch can run past `REQUEST_DEADLINE`.
Fix the comment, bound the write by the request deadline, and note in CONFIGS.md that REQUEST_TIMED_OUT leaves the stored retention unknown.
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -494,3 +534,108 @@ impl IggyBridge {
Ok(results)
}
}
+
+/// `message_expiry` and admission options from one `get_topics` row.
+///
+/// Enough for `DescribeConfigs`. The partition list on `TopicDetails` is not loaded.
+#[derive(Debug, Clone)]
+pub struct TopicConfigSnapshot {
Review Comment:
This keeps the whole `ResourceOptions` map for every topic, but describe only reads one flag from it (whether the expiry was set explicitly).
Store that `bool` at :622 and pass it to `described_entries`. It also removes about 35 lines of test setup.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
Review Comment:
A Kafka broker rejects `retention.minutes` and `retention.hours` on a topic. They only exist as broker keys (`log.retention.*`).
The synonym memory is also per process, so a restart or a second gateway loses it.
I think we need to treat both as unknown keys and drop `RetentionSynonymMemory` and its helpers. Keep `topic_identity` because `lookup` uses it.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
Review Comment:
`retention_ms` formats the millis as a string, and `retention_in_unit` (:255) parses that string back.
Keep the millis as a number and format once at the end. If the minutes/hours names go away, this code goes with them anyway.
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ DescribeConfigsResult::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(message.clone()),
+ Vec::new(),
+ )
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
Review Comment:
The 100-topic cap does not save much here because describe costs one `get_topics` per stream. It does break `kafka-topics.sh --describe`, which puts every topic in one request.
Can we keep the cap for AlterConfigs and CreateTopics only?
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -494,3 +534,108 @@ impl IggyBridge {
Ok(results)
}
}
+
+/// `message_expiry` and admission options from one `get_topics` row.
+///
+/// Enough for `DescribeConfigs`. The partition list on `TopicDetails` is not loaded.
+#[derive(Debug, Clone)]
+pub struct TopicConfigSnapshot {
+ pub message_expiry: IggyExpiry,
+ pub options: ResourceOptions,
+}
+
+/// One stream's `get_topics` result, shared by every Kafka name that resolves there.
+#[derive(Debug, Default)]
+pub struct StreamTopicCache {
+ streams: HashMap<String, StreamLoad>,
+}
+
+#[derive(Debug)]
+enum StreamLoad {
+ Ready(HashMap<String, TopicConfigSnapshot>),
+ Failed(BridgeError),
+ NotStarted,
Review Comment:
`NotStarted` and `TimedOut` both end up as REQUEST_TIMED_OUT in the two callers, and the `None` arm at :596 can never run.
Return `TimedOut` at :613 and drop both `NotStarted` variants.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
Review Comment:
No error that reaches `bridge_failure` can be `InvalidKafkaTopicName` (alter_configs.rs:227 checks the name first) or `SendLost` (only produce creates it).
Both arms can go.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
Review Comment:
`Unknown(String)` copies every unknown key name, and only a unit test reads the copy.
Drop unknown keys inside `listed_keys` after the empty check and remove the variant.
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
Review Comment:
This returns INVALID_REQUEST for unknown tagged fields. KIP-482 says that a receiver skips tags it does not know, and the other handlers here already do that.
Better to drop this check, the one at :240, both in alter_configs.rs (:120, :207), and the CONFIGS.md row for it.
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
Review Comment:
This deadline check repeats the one at the top of the next loop pass, and it also hides that pass's warn log.
Remove it here and at describe_configs.rs:223-225.
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
Review Comment:
create_topics.rs:222 still has its own `find_duplicate_names`, plus two tests for it.
CreateTopics can call this one with `topic.name.as_str()` and drop the copy, about 30 lines less.
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
Review Comment:
`describe_all` and `alter_all` (alter_configs.rs:114) repeat the same tagged-field, cap and duplicate checks, and each one builds its placeholder result twice.
One shared function in topic_config.rs can do these checks and return the distinct count too. Then `exceeds_topic_cap` can go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4179488236
##########
gateways/kafka/tests/configs_java_client_tests.rs:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Kafka 3.9 `AdminClient` describe/alter against the gateway, on the host JDK.
+//!
+//! The docker `--network host` suite cannot reach a loopback gateway from macOS.
+//! This test publishes nothing into a container: it binds `127.0.0.1` and runs
+//! `java` on the host. Jars come from Maven Central when they are not cached.
+
+use std::io::Read;
+use std::net::SocketAddr;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Output, Stdio};
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use tokio::sync::broadcast;
+
+use iggy_gateway_kafka::bridge::IggyBridge;
+use iggy_gateway_kafka::server::bind_listener;
+use iggy_gateway_kafka::{GatewayConfig, KafkaGateway};
+
+#[path = "common/iggy_server.rs"]
+mod iggy_server;
+
+use iggy_server::TestServer;
+
+const TOPIC: &str = "orders";
+
+/// Bounds a child the way `kafka_client_e2e_tests.rs` bounds a client run, without GNU `timeout`
+/// (this suite runs on the host JDK, including macOS). A hung `java` or `curl` used to block the
+/// process until the runner itself was killed.
+const CHILD_LIMIT: Duration = Duration::from_secs(120);
+
+/// Reports why the suite cannot run, and whether that is fatal.
+///
+/// Returns `true` when the caller should skip. `JAVA_E2E_REQUIRED=1` makes it panic instead, so a
+/// CI job that means to run this fails loudly rather than reporting a pass over zero assertions -
+/// same convention as `kafka_client_e2e_tests.rs`'s own `skip`/`KAFKA_E2E_REQUIRED`.
+fn skip(reason: &str) -> bool {
+ assert!(
+ std::env::var("JAVA_E2E_REQUIRED").as_deref() != Ok("1"),
+ "JAVA_E2E_REQUIRED=1 but the suite cannot run: {reason}"
+ );
+ eprintln!("skipping Java admin-client end-to-end test: {reason}");
+ true
+}
+
+fn java_missing() -> bool {
+ let available = Command::new("java")
+ .arg("-version")
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .status()
+ .is_ok_and(|status| status.success());
+ if available {
+ false
+ } else {
+ skip("java is unavailable")
+ }
+}
+
+fn output_bounded(command: &mut Command, what: &str) -> Output {
+ let mut child = command
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .unwrap_or_else(|error| panic!("spawn {what}: {error}"));
+ let mut stdout = child.stdout.take().expect("stdout");
+ let mut stderr = child.stderr.take().expect("stderr");
+ let stdout_thread = std::thread::spawn(move || {
+ let mut buf = Vec::new();
+ stdout.read_to_end(&mut buf).map(|_read| buf)
+ });
+ let stderr_thread = std::thread::spawn(move || {
+ let mut buf = Vec::new();
+ stderr.read_to_end(&mut buf).map(|_read| buf)
+ });
+ let deadline = Instant::now() + CHILD_LIMIT;
+ let status = loop {
+ match child.try_wait() {
+ Ok(Some(status)) => break status,
+ Ok(None) if Instant::now() >= deadline => {
+ let _ = child.kill();
+ let _ = child.wait();
+ panic!("{what} exceeded {CHILD_LIMIT:?}");
+ }
+ Ok(None) => std::thread::sleep(Duration::from_millis(50)),
+ Err(error) => panic!("poll {what}: {error}"),
+ }
+ };
+ let stdout = stdout_thread
+ .join()
+ .unwrap_or_else(|_| panic!("{what} stdout thread"))
+ .unwrap_or_else(|error| panic!("read {what} stdout: {error}"));
+ let stderr = stderr_thread
+ .join()
+ .unwrap_or_else(|_| panic!("{what} stderr thread"))
+ .unwrap_or_else(|error| panic!("read {what} stderr: {error}"));
+ Output {
+ status,
+ stdout,
+ stderr,
+ }
+}
+
+const JARS: &[(&str, &str)] = &[
+ (
+ "kafka-clients-3.9.0.jar",
+ "https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.9.0/kafka-clients-3.9.0.jar",
+ ),
+ (
+ "slf4j-api-1.7.36.jar",
+ "https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar",
+ ),
+ (
+ "slf4j-simple-1.7.36.jar",
+ "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/1.7.36/slf4j-simple-1.7.36.jar",
+ ),
+ (
+ "lz4-java-1.8.0.jar",
+ "https://repo1.maven.org/maven2/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar",
+ ),
+ (
+ "snappy-java-1.1.10.5.jar",
+ "https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.10.5/snappy-java-1.1.10.5.jar",
+ ),
+ (
+ "zstd-jni-1.5.6-4.jar",
+ "https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.5.6-4/zstd-jni-1.5.6-4.jar",
+ ),
+];
+
+fn jar_dir() -> PathBuf {
+ PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target/kafka-3.9-admin-jars")
+}
+
+fn ensure_jars(dir: &Path) {
+ std::fs::create_dir_all(dir).expect("create jar cache");
+ for (name, url) in JARS {
+ let path = dir.join(name);
+ if path.exists() {
+ continue;
+ }
+ let output = output_bounded(
+ Command::new("curl")
+ .args([
+ "--fail",
+ "--silent",
+ "--show-error",
+ "--location",
+ "--max-time",
+ "60",
+ "-o",
+ ])
+ .arg(&path)
+ .arg(url),
+ name,
+ );
+ if !output.status.success() {
+ let _ = std::fs::remove_file(&path);
+ }
+ assert!(
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,725 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+/// Kafka `ConfigType.UNKNOWN`.
+pub const CONFIG_TYPE_UNKNOWN: i8 = 0;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct Kafka topic names one `DescribeConfigs` or `AlterConfigs` request may send to Iggy.
+///
+/// Same ceiling as `CreateTopics`: each distinct name is a lockstep call on the shared client.
+/// Duplicate names in one request count once for this cap, but a repeated name is itself
+/// rejected outright - see [`find_duplicate_names`]. A larger batch is `POLICY_VIOLATION` on
+/// every resource. The decode-time element ceiling is a separate, larger limit and is not this
+/// cap.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`] and stays in the list.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
+ let Some(keys) = configuration_keys.filter(|keys| !keys.is_empty()) else {
+ return vec![ListedKey::Retention, ListedKey::Cleanup];
+ };
+ keys.iter()
+ .map(|key| match key.as_str() {
+ RETENTION_MS => ListedKey::Retention,
+ RETENTION_MINUTES => ListedKey::Minutes,
+ RETENTION_HOURS => ListedKey::Hours,
+ CLEANUP_POLICY => ListedKey::Cleanup,
+ other => ListedKey::Unknown(other.to_string()),
+ })
+ .collect()
+}
+
+/// The expiry to store, or `None` when the resource names no retention key.
+///
+/// `retention.ms`, `retention.minutes`, and `retention.hours` are one duration.
+/// Minutes and hours convert to milliseconds. Two names that convert to
+/// different millisecond values fail the resource and select nothing. One
+/// expiry is returned when they agree. [`PlannedRetention::synonyms`] is empty
+/// when the resource used only `retention.ms`.
+///
+/// One invalid or repeated key fails the whole resource. Nothing is selected
+/// for storage in that case. `update_topic` stores the millisecond expiry, never
+/// the minute or hour count.
+///
+/// # Errors
+///
+/// Returns the first [`ConfigFault`] in request order.
+pub fn plan_retention_update<'a>(
+ configs: impl IntoIterator<Item = (&'a str, Option<&'a str>)>,
+) -> Result<Option<PlannedRetention>, ConfigFault> {
+ let mut seen = HashSet::new();
+ let mut expiry = None;
+ let mut synonyms = RetentionSynonyms::default();
+ for (name, value) in configs {
+ if !seen.insert(name) {
+ return Err(ConfigFault::Repeated(name.to_string()));
+ }
+ let unit_ms = retention_scale(name, &mut synonyms)?;
+ let Some(value) = value else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ let parsed = if unit_ms == 1 {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205827844
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205823754
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205825681
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4179489387
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,725 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+/// Kafka `ConfigType.UNKNOWN`.
+pub const CONFIG_TYPE_UNKNOWN: i8 = 0;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct Kafka topic names one `DescribeConfigs` or `AlterConfigs` request may send to Iggy.
+///
+/// Same ceiling as `CreateTopics`: each distinct name is a lockstep call on the shared client.
+/// Duplicate names in one request count once for this cap, but a repeated name is itself
+/// rejected outright - see [`find_duplicate_names`]. A larger batch is `POLICY_VIOLATION` on
+/// every resource. The decode-time element ceiling is a separate, larger limit and is not this
+/// cap.
+pub const MAX_CONFIG_TOPICS: usize = 100;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4398:
URL: https://github.com/apache/iggy/pull/4398#issuecomment-5984854261
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4398:
URL: https://github.com/apache/iggy/pull/4398#issuecomment-5980287684
Keeping it in draft mode as there few edge cases needs to be worked and documented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205818814
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205908043
##########
gateways/kafka/README.md:
##########
@@ -1,11 +1,12 @@
# Kafka gateway (`iggy-gateway-kafka`)
-Foundation layer for [apache/iggy#3421](https://github.com/apache/iggy/issues/3421): a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata and CreateTopics use Iggy. InitProducerId and consumer group coordination work with or without one.
+Foundation layer for [apache/iggy#3421](https://github.com/apache/iggy/issues/3421): a TCP listener on the Kafka wire port that decodes requests and validates scoped API keys and versions. With a bridge, Produce, Fetch, ListOffsets, Metadata, CreateTopics, DescribeConfigs and AlterConfigs use Iggy. InitProducerId and consumer group coordination work with or without one.
-> **Stub warning:** with `IGGY_KAFKA_BRIDGE_ENABLED=true`, Produce, Fetch, ListOffsets, Metadata and
-> CreateTopics use Iggy. With the bridge off (the default), those five are stubs: Produce, Fetch and
-> ListOffsets answer retriable `NOT_LEADER_OR_FOLLOWER` (6), CreateTopics answers `NOT_CONTROLLER`
-> (41), and Metadata reports every requested topic unknown. **CreateTopics runs as the bridge's own
+> **Stub warning:** with `IGGY_KAFKA_BRIDGE_ENABLED=true`, Produce, Fetch, ListOffsets, Metadata,
+> CreateTopics, DescribeConfigs and AlterConfigs use Iggy. With the bridge off (the default), those
+> seven are stubs: Produce, Fetch and ListOffsets answer retriable `NOT_LEADER_OR_FOLLOWER` (6),
+> CreateTopics, DescribeConfigs and AlterConfigs answer `NOT_CONTROLLER` (41), and Metadata reports
+> every requested topic unknown. **CreateTopics runs as the bridge's own
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &DescribeConfigsRequest,
+ deadline: Instant,
+) -> Vec<DescribeConfigsResult> {
+ if !req.unknown_tagged_fields.is_empty() {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4398:
URL: https://github.com/apache/iggy/pull/4398#issuecomment-5984166671
/skill team-review-slim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4398:
URL: https://github.com/apache/iggy/pull/4398#issuecomment-6050139966
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio merged PR #4398:
URL: https://github.com/apache/iggy/pull/4398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205917605
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205904255
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`]. `DescribeConfigs` omits
+/// those names from the response and does not fail the resource.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`]. `DescribeConfigs` omits
+/// those names from the response and does not fail the resource.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205921443
##########
gateways/kafka/src/protocol/handlers/describe_configs.rs:
##########
@@ -0,0 +1,693 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DescribeConfigs` (API key 32).
+
+use bytes::Bytes;
+use iggy::prelude::{IggyExpiry, ResourceOptions};
+use kafka_protocol::messages::DescribeConfigsRequest;
+use kafka_protocol::messages::describe_configs_request::DescribeConfigsResource;
+use kafka_protocol::messages::describe_configs_response::{
+ DescribeConfigsResourceResult, DescribeConfigsResponse, DescribeConfigsResult,
+ DescribeConfigsSynonym,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_DESCRIBE_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_describe_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ CLEANUP_DOC, CLEANUP_POLICY, CLEANUP_POLICY_VALUE, CONFIG_SOURCE_DEFAULT, CONFIG_SOURCE_TOPIC,
+ CONFIG_TYPE_LONG, CONFIG_TYPE_STRING, IggyTopicKey, ListedKey, MILLIS_PER_HOUR,
+ MILLIS_PER_MINUTE, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, RETENTION_DOC, RETENTION_HOURS,
+ RETENTION_HOURS_DOC, RETENTION_MINUTES, RETENTION_MINUTES_DOC, RETENTION_MS, RetentionMs,
+ RetentionSynonyms, bridge_failure, exceeds_topic_cap, find_duplicate_names, listed_keys,
+ message_expiry_is_explicit, name_reason, retention_in_unit, retention_ms, static_text,
+ topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DESCRIBE_CONFIGS,
+ min_version: 1,
+ max_version: 4,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_DESCRIBE_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_DESCRIBE_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DescribeConfigsRequest>(api_version, body, |version, body| {
+ validate_describe_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode DescribeConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DescribeConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let results = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None, Vec::new()))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let results = describe_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(results), api_version, 256),
+ "DescribeConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let results = vec![
+ DescribeConfigsResult::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str(""))
+ .with_configs(Vec::new()),
+ ];
+ encode_message(&response(results), version, 128)
+}
+
+fn response(results: Vec<DescribeConfigsResult>) -> DescribeConfigsResponse {
+ DescribeConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_results(results)
+}
+
+async fn describe_all(
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205915500
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut cache = StreamTopicCache::default();
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(
+ state,
+ bridge,
+ &mut cache,
+ resource,
+ req.validate_only,
+ deadline,
+ )
+ .await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ cache: &mut StreamTopicCache,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ match cache.lookup(bridge, name, deadline, "AlterConfigs").await {
+ TopicLoad::Found(_) => {}
+ TopicLoad::Missing => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ TopicLoad::Failed(error) => {
+ let (code, message) = bridge_failure(error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ TopicLoad::NotStarted | TopicLoad::TimedOut => {
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // Kafka's AlterConfigs replaces the whole set, so an empty list is a reset.
+ // This gateway patches named keys. Reporting success would hide that the
+ // previous expiry is still stored.
+ let Some(planned) = planned else {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(static_text(
+ "an empty configs list is rejected because this gateway patches named keys and does not replace the topic configuration",
+ )),
+ );
+ };
+
+ // `validate_only` already performed the existence read and the same key checks.
+ // It does not call `update_topic` and does not change synonym memory.
+ if validate_only {
+ return resource_error(resource, ERROR_NONE, None);
+ }
+
+ // Do not start the write once the deadline has passed, and do not wrap it in
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,325 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge, StreamTopicCache, TopicLoad};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4398:
URL: https://github.com/apache/iggy/pull/4398#issuecomment-5980854760
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4398?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 28.14%. Comparing base ([`2d7fddb`](https://app.codecov.io/gh/apache/iggy/commit/2d7fddba5552640518aa7f1372f69af9c77e436a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e830007`](https://app.codecov.io/gh/apache/iggy/commit/e8300074b6a16a8b016cdcaffe54cebec42428c9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4398 +/- ##
=============================================
- Coverage 87.96% 28.14% -59.82%
Complexity 1579 1579
=============================================
Files 1290 1287 -3
Lines 230145 183374 -46771
Branches 193480 146709 -46771
=============================================
- Hits 202440 51609 -150831
- Misses 22980 129930 +106950
+ Partials 4725 1835 -2890
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `13.35% <ø> (-75.72%)` | :arrow_down: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.74% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.62% <ø> (ø)` | |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `91.24% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.67% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.59% <ø> (ø)` | |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4398/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.19% <ø> (ø)` | |
[see 693 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4398/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4179486480
##########
gateways/kafka/src/protocol/handlers/alter_configs.rs:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `AlterConfigs` (API key 33).
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::AlterConfigsRequest;
+use kafka_protocol::messages::alter_configs_request::AlterConfigsResource;
+use kafka_protocol::messages::alter_configs_response::{
+ AlterConfigsResourceResponse, AlterConfigsResponse,
+};
+use kafka_protocol::protocol::StrBytes;
+use tokio::time::Instant;
+
+use crate::bridge::topic_map::validate_kafka_topic_name;
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::protocol::api::{
+ API_KEY_ALTER_CONFIGS, ApiVersionRange, ERROR_INVALID_CONFIG, ERROR_INVALID_REQUEST,
+ ERROR_INVALID_TOPIC_EXCEPTION, ERROR_NONE, ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION,
+ ERROR_REQUEST_TIMED_OUT, ERROR_UNKNOWN_TOPIC_OR_PARTITION, GatewayState, HandleOutcome,
+ REQUEST_DEADLINE,
+};
+use crate::protocol::bounds_guard::validate_alter_configs_shape;
+use crate::protocol::handlers::topic_config::{
+ IggyTopicKey, ONLY_TOPIC_RESOURCES, RESOURCE_TYPE_TOPIC, bridge_failure, exceeds_topic_cap,
+ find_duplicate_names, name_reason, plan_retention_update, static_text, topic_cap_message,
+};
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, is_supported_version, respond_or_close,
+ unsupported_version_response,
+};
+
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_ALTER_CONFIGS,
+ min_version: 0,
+ max_version: 2,
+};
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ if !is_supported_version(API_KEY_ALTER_CONFIGS, api_version) {
+ return unsupported_version_response(API_KEY_ALTER_CONFIGS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<AlterConfigsRequest>(api_version, body, |version, body| {
+ validate_alter_configs_shape(version, body, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ tracing::debug!(%error, "Failed to decode AlterConfigs request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "AlterConfigs",
+ );
+ }
+ };
+
+ let Some(bridge) = &state.bridge else {
+ let responses = req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_NOT_CONTROLLER, None))
+ .collect();
+ return respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ );
+ };
+
+ let deadline = Instant::now() + REQUEST_DEADLINE;
+ let responses = alter_all(state, bridge, &req, deadline).await;
+ respond_or_close(
+ encode_message(&response(responses), api_version, 256),
+ "AlterConfigs",
+ )
+}
+
+/// # Errors
+///
+/// Returns an error when `kafka_protocol` cannot encode the response at `version`.
+pub fn encode_error_response(version: i16, error_code: i16) -> crate::error::Result<Bytes> {
+ let responses = vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(error_code)
+ .with_error_message(None)
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ];
+ encode_message(&response(responses), version, 128)
+}
+
+fn response(responses: Vec<AlterConfigsResourceResponse>) -> AlterConfigsResponse {
+ AlterConfigsResponse::default()
+ .with_throttle_time_ms(0)
+ .with_responses(responses)
+}
+
+async fn alter_all(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ req: &AlterConfigsRequest,
+ deadline: Instant,
+) -> Vec<AlterConfigsResourceResponse> {
+ if !req.unknown_tagged_fields.is_empty() {
+ let message = static_text("unknown tagged fields are not supported");
+ return if req.resources.is_empty() {
+ vec![
+ AlterConfigsResourceResponse::default()
+ .with_error_code(ERROR_INVALID_REQUEST)
+ .with_error_message(Some(message))
+ .with_resource_type(RESOURCE_TYPE_TOPIC)
+ .with_resource_name(StrBytes::from_static_str("")),
+ ]
+ } else {
+ req.resources
+ .iter()
+ .map(|resource| {
+ resource_error(resource, ERROR_INVALID_REQUEST, Some(message.clone()))
+ })
+ .collect()
+ };
+ }
+
+ let topic_names: Vec<&str> = req
+ .resources
+ .iter()
+ .filter(|resource| resource.resource_type == RESOURCE_TYPE_TOPIC)
+ .map(|resource| resource.resource_name.as_str())
+ .collect();
+ if exceeds_topic_cap(topic_names.iter().copied()) {
+ let message = StrBytes::from(topic_cap_message("AlterConfigs"));
+ tracing::warn!("AlterConfigs request addresses too many distinct topics; rejecting");
+ return req
+ .resources
+ .iter()
+ .map(|resource| resource_error(resource, ERROR_POLICY_VIOLATION, Some(message.clone())))
+ .collect();
+ }
+
+ // Real Kafka refuses every occurrence of a duplicate resource name outright rather than
+ // picking a winner - the same choice CreateTopics makes for a repeated topic name.
+ let duplicate_names = find_duplicate_names(topic_names.iter().copied());
+
+ let mut responses = Vec::with_capacity(req.resources.len());
+ let mut deadline_exceeded = false;
+ for resource in &req.resources {
+ if resource.resource_type == RESOURCE_TYPE_TOPIC
+ && duplicate_names.contains(resource.resource_name.as_str())
+ {
+ responses.push(resource_error(resource, ERROR_INVALID_REQUEST, None));
+ continue;
+ }
+ if deadline_exceeded || Instant::now() >= deadline {
+ if !deadline_exceeded {
+ deadline_exceeded = true;
+ tracing::warn!(
+ "AlterConfigs deadline passed; answering remaining resources retriable \
+ instead of starting new Iggy calls"
+ );
+ }
+ responses.push(resource_error(resource, ERROR_REQUEST_TIMED_OUT, None));
+ continue;
+ }
+
+ let result = alter_one(state, bridge, resource, req.validate_only, deadline).await;
+ if Instant::now() >= deadline {
+ deadline_exceeded = true;
+ }
+ responses.push(result);
+ }
+ responses
+}
+
+async fn alter_one(
+ state: &GatewayState,
+ bridge: &IggyBridge,
+ resource: &AlterConfigsResource,
+ validate_only: bool,
+ deadline: Instant,
+) -> AlterConfigsResourceResponse {
+ if !resource.unknown_tagged_fields.is_empty()
+ || resource
+ .configs
+ .iter()
+ .any(|config| !config.unknown_tagged_fields.is_empty())
+ {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text("unknown tagged fields are not supported")),
+ );
+ }
+ if resource.resource_type != RESOURCE_TYPE_TOPIC {
+ return resource_error(
+ resource,
+ ERROR_INVALID_REQUEST,
+ Some(static_text(ONLY_TOPIC_RESOURCES)),
+ );
+ }
+ let name = resource.resource_name.as_str();
+ if let Err(error) = validate_kafka_topic_name("kafka_topic", name) {
+ return resource_error(
+ resource,
+ ERROR_INVALID_TOPIC_EXCEPTION,
+ Some(StrBytes::from(name_reason(&error))),
+ );
+ }
+
+ // Existence is checked before config keys. A missing topic is
+ // UNKNOWN_TOPIC_OR_PARTITION even when a key would also be invalid.
+ // A deadline that has already passed must not start the lockstep Iggy call.
+ if Instant::now() >= deadline {
+ tracing::warn!(
+ resource = name,
+ "AlterConfigs deadline passed before the topic lookup"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ match tokio::time::timeout_at(deadline, bridge.get_kafka_topic(name)).await {
+ Ok(Ok(Some(_topic))) => {}
+ Ok(Ok(None)) => {
+ return resource_error(resource, ERROR_UNKNOWN_TOPIC_OR_PARTITION, None);
+ }
+ Ok(Err(error)) => {
+ let (code, message) = bridge_failure(&error, "AlterConfigs", "altering");
+ return resource_error(resource, code, message);
+ }
+ Err(_elapsed) => {
+ tracing::warn!(
+ resource = name,
+ "AlterConfigs: this resource's bridge work exceeded the request deadline; \
+ answering retriable instead of blocking further"
+ );
+ return resource_error(resource, ERROR_REQUEST_TIMED_OUT, None);
+ }
+ }
+
+ let planned = match plan_retention_update(resource.configs.iter().map(|config| {
+ (
+ config.name.as_str(),
+ config.value.as_ref().map(StrBytes::as_str),
+ )
+ })) {
+ Ok(planned) => planned,
+ Err(fault) => {
+ return resource_error(
+ resource,
+ ERROR_INVALID_CONFIG,
+ Some(StrBytes::from(fault.message())),
+ );
+ }
+ };
+ // A resource that names no retention key has nothing to store. Omitting the key
+ // does not clear a previously set expiry or the remembered synonym names.
+ let Some(planned) = planned else {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205821265
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205817223
##########
gateways/kafka/docs/CONFIGS.md:
##########
@@ -0,0 +1,131 @@
+# Topic configuration
+
+`DescribeConfigs` (32) and `AlterConfigs` (33) apply to Kafka topic resources only
Review Comment:
fixed
##########
gateways/kafka/src/protocol/bounds_guard.rs:
##########
@@ -1052,6 +1052,110 @@ pub fn validate_sync_group_shape(version: i16, body: &Bytes, max_frame_size: usi
/// reaches `parse_plain`, and what a future mechanism would carry into a credential exchange.
const MAX_SASL_AUTH_BYTES: usize = 4096;
+/// Mirrors the field order `DescribeConfigsRequest::decode` walks.
+///
+/// # Errors
+///
+/// Returns an error when a declared array or string length cannot fit in the bytes remaining
+/// in the frame, or the body is truncated or malformed in a way that cannot be walked.
+pub fn validate_describe_configs_shape(
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
+ names.into_iter().collect::<HashSet<_>>().len() > MAX_CONFIG_TOPICS
+}
+
+/// Client-facing text for a request over [`MAX_CONFIG_TOPICS`].
+#[must_use]
+pub fn topic_cap_message(api_name: &str) -> String {
+ format!(
+ "this gateway addresses at most {MAX_CONFIG_TOPICS} distinct topics per {api_name} request"
+ )
+}
+
+/// Names in `names` that occur more than once.
+///
+/// Real Kafka refuses every occurrence of a duplicate resource name with `INVALID_REQUEST`
+/// (42) rather than silently picking a winner - the same choice `CreateTopics`' own
+/// `find_duplicate_names` makes for a repeated topic name in one batch.
+#[must_use]
+pub fn find_duplicate_names<'a>(names: impl IntoIterator<Item = &'a str>) -> HashSet<&'a str> {
+ let mut seen = HashSet::new();
+ let mut duplicates = HashSet::new();
+ for name in names {
+ if !seen.insert(name) {
+ duplicates.insert(name);
+ }
+ }
+ duplicates
+}
+
+/// The reason a topic name failed Kafka's own naming rules, or a fixed fallback.
+#[must_use]
+pub fn name_reason(error: &BridgeError) -> String {
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => reason.clone(),
+ _ => "invalid topic name".to_string(),
+ }
+}
+
+#[must_use]
+pub const fn static_text(message: &'static str) -> StrBytes {
+ StrBytes::from_static_str(message)
+}
+
+/// Maps a bridge failure to a Kafka error code and client-facing message, logging the real
+/// cause server-side.
+///
+/// `handler` names the API in the log line (`"DescribeConfigs"`/`"AlterConfigs"`); `action` is
+/// the present participle of what the bridge call was doing (`"reading"`/`"altering"`), reused
+/// in both the timeout log and the generic internal-error message.
+#[must_use]
+pub fn bridge_failure(error: &BridgeError, handler: &str, action: &str) -> (i16, Option<StrBytes>) {
+ let code = error.to_kafka_error_code();
+ match error {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(reason, "{handler} rejected an invalid topic name");
+ (code, Some(StrBytes::from(reason.clone())))
+ }
+ BridgeError::Timeout | BridgeError::SendLost(_) => {
+ tracing::warn!(%error, "{handler} {action} exceeded the bridge deadline");
+ (code, None)
+ }
+ other => {
+ tracing::error!(%other, "{handler} failed while {action} topic configuration");
+ (
+ code,
+ Some(StrBytes::from(format!(
+ "internal error {action} topic configuration"
+ ))),
+ )
+ }
+ }
+}
+
+/// Keys to return for one describe resource.
+///
+/// `None` and an empty list both mean `retention.ms` and `cleanup.policy`.
+/// `retention.minutes` and `retention.hours` are listed only when the client
+/// names them. Any other name is [`ListedKey::Unknown`]. `DescribeConfigs` omits
+/// those names from the response and does not fail the resource.
+#[must_use]
+pub fn listed_keys(configuration_keys: Option<&[StrBytes]>) -> Vec<ListedKey> {
+ let Some(keys) = configuration_keys.filter(|keys| !keys.is_empty()) else {
+ return vec![ListedKey::Retention, ListedKey::Cleanup];
+ };
+ keys.iter()
+ .map(|key| match key.as_str() {
+ RETENTION_MS => ListedKey::Retention,
+ RETENTION_MINUTES => ListedKey::Minutes,
+ RETENTION_HOURS => ListedKey::Hours,
+ CLEANUP_POLICY => ListedKey::Cleanup,
+ other => ListedKey::Unknown(other.to_string()),
+ })
+ .collect()
+}
+
+/// The expiry to store, or `None` when the resource names no retention key.
+///
+/// `retention.ms`, `retention.minutes`, and `retention.hours` are one duration.
+/// Minutes and hours convert to milliseconds. Two names that convert to
+/// different millisecond values fail the resource and select nothing. One
+/// expiry is returned when they agree. [`PlannedRetention::synonyms`] is empty
+/// when the resource used only `retention.ms`.
+///
+/// One invalid or repeated key fails the whole resource. Nothing is selected
+/// for storage in that case. `update_topic` stores the millisecond expiry, never
+/// the minute or hour count.
+///
+/// # Errors
+///
+/// Returns the first [`ConfigFault`] in request order.
+pub fn plan_retention_update<'a>(
+ configs: impl IntoIterator<Item = (&'a str, Option<&'a str>)>,
+) -> Result<Option<PlannedRetention>, ConfigFault> {
+ let mut seen = HashSet::new();
+ let mut expiry = None;
+ let mut synonyms = RetentionSynonyms::default();
+ for (name, value) in configs {
+ if !seen.insert(name) {
+ return Err(ConfigFault::Repeated(name.to_string()));
+ }
+ let unit_ms = retention_scale(name, &mut synonyms)?;
+ let Some(value) = value else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ let parsed = parse_scaled_retention(value, unit_ms)?;
+ if let Some(existing) = expiry
+ && existing != parsed
+ {
+ return Err(ConfigFault::ConflictingRetention);
+ }
+ expiry = Some(parsed);
+ }
+ Ok(expiry.map(|expiry| PlannedRetention { expiry, synonyms }))
+}
+
+fn retention_scale(name: &str, synonyms: &mut RetentionSynonyms) -> Result<u64, ConfigFault> {
Review Comment:
fxed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DescribeConfigs and AlterConfigs APIs [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4398:
URL: https://github.com/apache/iggy/pull/4398#discussion_r4205815237
##########
gateways/kafka/docs/CONFIGS.md:
##########
@@ -0,0 +1,131 @@
+# Topic configuration
+
+`DescribeConfigs` (32) and `AlterConfigs` (33) apply to Kafka topic resources only
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/topic_config.rs:
##########
@@ -0,0 +1,721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Topic configuration shared by `DescribeConfigs` and `AlterConfigs`.
+//!
+//! Iggy stores `message_expiry` as [`IggyExpiry`]. A duration is microseconds
+//! (`IggyDuration::from(u64)` / `as_micros`). `u64::MAX` is [`IggyExpiry::NeverExpire`]
+//! and `0` is [`IggyExpiry::ServerDefault`], so neither is a usable millisecond expiry.
+//! Kafka `retention.ms` is a decimal millisecond count, or `-1` when sealed segments
+//! never expire. `retention.minutes` and `retention.hours` are the same duration,
+//! converted to milliseconds before anything is stored. A positive count is capped
+//! at `u32::MAX` seconds, matching [`IggyExpiry::from_str`].
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use iggy::prelude::{HeaderKey, IggyDuration, IggyExpiry, ResourceOptions, topic_option_keys};
+use kafka_protocol::protocol::StrBytes;
+
+use crate::bridge::BridgeError;
+
+/// Kafka `ConfigResource.Type.TOPIC`.
+pub const RESOURCE_TYPE_TOPIC: i8 = 2;
+
+/// Kafka `ConfigSource.DYNAMIC_TOPIC_CONFIG`: the value was stored on the topic.
+pub const CONFIG_SOURCE_TOPIC: i8 = 1;
+
+/// Kafka `ConfigSource.DEFAULT_CONFIG`: the value is the gateway default.
+pub const CONFIG_SOURCE_DEFAULT: i8 = 5;
+
+/// Kafka `ConfigType.STRING`.
+pub const CONFIG_TYPE_STRING: i8 = 2;
+
+/// Kafka `ConfigType.LONG`.
+pub const CONFIG_TYPE_LONG: i8 = 5;
+
+pub const RETENTION_MS: &str = "retention.ms";
+pub const RETENTION_MINUTES: &str = "retention.minutes";
+pub const RETENTION_HOURS: &str = "retention.hours";
+pub const MILLIS_PER_MINUTE: u64 = 60_000;
+pub const MILLIS_PER_HOUR: u64 = 3_600_000;
+pub const CLEANUP_POLICY: &str = "cleanup.policy";
+pub const CLEANUP_POLICY_VALUE: &str = "delete";
+
+pub const ONLY_TOPIC_RESOURCES: &str = "only topic resources are supported";
+
+pub const RETENTION_DOC: &str = "How long a sealed segment is kept, in milliseconds. The active segment does not expire. -1 means sealed segments never expire.";
+pub const RETENTION_MINUTES_DOC: &str = "How long a sealed segment is kept, in minutes, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const RETENTION_HOURS_DOC: &str = "How long a sealed segment is kept, in hours, derived from retention.ms. The count truncates toward zero. -1 means sealed segments never expire.";
+pub const CLEANUP_DOC: &str = "Iggy deletes expired messages and does not compact a topic.";
+
+/// Distinct topic names one `CreateTopics`, `DescribeConfigs`, or `AlterConfigs` request
+/// may address through the bridge.
+///
+/// `bounds_guard`'s element ceiling is a pre-decode limit. `CreateTopics` can spend several
+/// lockstep Iggy calls per name on the shared client (`ensure_stream`, `create_topic`, and a
+/// possible race-retry read). 100 keeps that worst case small and is shared with the config
+/// APIs. Duplicate names count once toward the cap and are rejected on their own. A larger
+/// batch is `POLICY_VIOLATION` on every resource.
+pub const MAX_CONFIG_TOPICS: usize = 100;
+
+/// One config name a describe request asked for, in request order.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ListedKey {
+ Retention,
+ Minutes,
+ Hours,
+ Cleanup,
+ Unknown(String),
+}
+
+/// Why one resource's alter cannot be stored.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ConfigFault {
+ UnknownKey(String),
+ Repeated(String),
+ ReadOnly(&'static str),
+ InvalidRetention,
+ ConflictingRetention,
+}
+
+impl ConfigFault {
+ #[must_use]
+ pub fn message(&self) -> String {
+ match self {
+ Self::UnknownKey(name) => format!("unknown config key '{name}'"),
+ Self::Repeated(name) => format!("config key '{name}' is repeated"),
+ Self::ReadOnly(message) => (*message).to_string(),
+ Self::InvalidRetention => format!(
+ "retention.ms, retention.minutes, and retention.hours must be -1 or a positive count whose duration is at most {} seconds",
+ u32::MAX
+ ),
+ Self::ConflictingRetention => "retention.ms, retention.minutes, and retention.hours must describe the same duration".to_string(),
+ }
+ }
+}
+
+/// `retention.minutes` and `retention.hours` used by the last successful alter.
+///
+/// Process-local, keyed by Iggy stream and topic, and not stored in Iggy. A
+/// restart clears it. Deleting the topic outside this gateway does not.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct RetentionSynonyms {
+ pub minutes: bool,
+ pub hours: bool,
+}
+
+impl RetentionSynonyms {
+ #[must_use]
+ pub const fn is_empty(self) -> bool {
+ !self.minutes && !self.hours
+ }
+}
+
+/// Iggy stream and topic a Kafka name resolves to. Synonym memory uses this pair,
+/// not the Kafka topic name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct IggyTopicKey {
+ stream: String,
+ topic: String,
+}
+
+impl IggyTopicKey {
+ #[must_use]
+ pub fn new(stream: impl Into<String>, topic: impl Into<String>) -> Self {
+ Self {
+ stream: stream.into(),
+ topic: topic.into(),
+ }
+ }
+}
+
+/// Process memory of [`RetentionSynonyms`] per [`IggyTopicKey`].
+#[derive(Debug, Default)]
+pub struct RetentionSynonymMemory {
+ topics: HashMap<IggyTopicKey, RetentionSynonyms>,
+}
+
+impl RetentionSynonymMemory {
+ #[must_use]
+ pub fn remembered(&self, key: &IggyTopicKey) -> RetentionSynonyms {
+ self.topics.get(key).copied().unwrap_or_default()
+ }
+
+ /// Remembers `synonyms` after a successful alter that stored an expiry.
+ ///
+ /// An empty set clears the key. That is an alter which used only
+ /// `retention.ms`. A failed alter, and a resource that stored nothing,
+ /// must not call this.
+ pub fn record_applied(&mut self, key: IggyTopicKey, synonyms: RetentionSynonyms) {
+ if synonyms.is_empty() {
+ self.topics.remove(&key);
+ } else {
+ self.topics.insert(key, synonyms);
+ }
+ }
+}
+
+/// One duration to store, plus which synonym names the alter used.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PlannedRetention {
+ pub expiry: IggyExpiry,
+ pub synonyms: RetentionSynonyms,
+}
+
+/// `retention.ms` as Kafka should report it, plus the config source byte.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetentionMs {
+ pub value: String,
+ pub source: i8,
+}
+
+/// Whether the topic's `message_expiry` option was sent by a client.
+///
+/// Admission stores the never-expire default with `explicit == false`. An alter
+/// of `retention.ms` stores the key with `explicit == true`, including `-1`.
+#[must_use]
+pub fn message_expiry_is_explicit(options: &ResourceOptions) -> bool {
+ let Ok(key) = HeaderKey::from_str(topic_option_keys::MESSAGE_EXPIRY) else {
+ return false;
+ };
+ options.get(&key).is_some_and(|entry| entry.explicit)
+}
+
+/// Map an Iggy expiry onto Kafka `retention.ms`.
+///
+/// A derived never-expire value is [`CONFIG_SOURCE_DEFAULT`]. An explicit one,
+/// and any stored duration, is [`CONFIG_SOURCE_TOPIC`]. A duration that is not a
+/// positive whole number of milliseconds is refused: reporting a rounded count
+/// would describe a different value than Iggy stored.
+///
+/// # Errors
+///
+/// Returns `Err` when `expiry` is a duration of `0` microseconds or a duration
+/// that is not a whole number of milliseconds.
+pub fn retention_ms(expiry: IggyExpiry, explicit: bool) -> Result<RetentionMs, ()> {
+ match expiry {
+ IggyExpiry::NeverExpire | IggyExpiry::ServerDefault => Ok(RetentionMs {
+ value: "-1".to_string(),
+ source: if explicit {
+ CONFIG_SOURCE_TOPIC
+ } else {
+ CONFIG_SOURCE_DEFAULT
+ },
+ }),
+ IggyExpiry::ExpireDuration(duration) => {
+ let micros = duration.as_micros();
+ if micros == 0 || !micros.is_multiple_of(1_000) {
+ return Err(());
+ }
+ Ok(RetentionMs {
+ value: (micros / 1_000).to_string(),
+ source: CONFIG_SOURCE_TOPIC,
+ })
+ }
+ }
+}
+
+/// Parse a Kafka `retention.ms` value into the expiry `update_topic` stores.
+///
+/// `-1` is [`IggyExpiry::NeverExpire`] (`u64::MAX` on the wire), not
+/// [`IggyExpiry::ServerDefault`] (`0`), because the server ignores the `0`
+/// sentinel and leaves the previous expiry in place. The text must be the
+/// canonical decimal of the parsed integer, so a later describe reports the
+/// same characters the alter accepted.
+///
+/// # Errors
+///
+/// Returns [`ConfigFault::InvalidRetention`] for any value other than `-1` or a
+/// positive millisecond count of at most `u32::MAX` seconds.
+#[cfg(test)]
+pub fn parse_retention_ms(value: &str) -> Result<IggyExpiry, ConfigFault> {
+ parse_scaled_retention(value, 1)
+}
+
+/// `retention.ms` text scaled into `unit_ms` units.
+///
+/// `-1` stays `-1`. Any other count is integer division toward zero, so a stored
+/// `90000` reported as minutes is `1` while `retention.ms` stays `90000`.
+#[must_use]
+pub fn retention_in_unit(millis_text: &str, unit_ms: u64) -> Option<String> {
+ let Ok(millis) = millis_text.parse::<i64>() else {
+ return None;
+ };
+ if millis == -1 {
+ return Some("-1".to_string());
+ }
+ if millis < 0 || unit_ms == 0 {
+ return None;
+ }
+ let Ok(millis) = u64::try_from(millis) else {
+ return None;
+ };
+ Some((millis / unit_ms).to_string())
+}
+
+/// `count` is one unit of `unit_ms` milliseconds. `1` is `retention.ms`.
+fn parse_scaled_retention(value: &str, unit_ms: u64) -> Result<IggyExpiry, ConfigFault> {
+ let Ok(count) = value.parse::<i64>() else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ if count.to_string() != value {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ if count == -1 {
+ return Ok(IggyExpiry::NeverExpire);
+ }
+ if count <= 0 {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let count = u64::try_from(count).map_err(|_| ConfigFault::InvalidRetention)?;
+ // Reject a product that does not fit before multiplying into milliseconds.
+ if unit_ms == 0 || count > u64::MAX / unit_ms {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let millis = count * unit_ms;
+ // `IggyExpiry::from_str` rejects `as_secs() > u32::MAX`. The typed update path
+ // does not, but a value the server's own parser refuses is not one this gateway
+ // should store.
+ if millis / 1_000 > u64::from(u32::MAX) {
+ return Err(ConfigFault::InvalidRetention);
+ }
+ let Some(micros) = millis.checked_mul(1_000) else {
+ return Err(ConfigFault::InvalidRetention);
+ };
+ Ok(IggyExpiry::ExpireDuration(IggyDuration::from(micros)))
+}
+
+/// Whether `names` contains more than [`MAX_CONFIG_TOPICS`] distinct strings.
+#[must_use]
+pub fn exceeds_topic_cap<'a>(names: impl IntoIterator<Item = &'a str>) -> bool {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org