You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ryerraguntla (via GitHub)" <gi...@apache.org> on 2026/10/04 16:29:19 UTC
[PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
ryerraguntla opened a new pull request, #4400:
URL: https://github.com/apache/iggy/pull/4400
## Which issue does this PR address?
Relates to #3546
## Rationale
Kafka clients remove a topic with DeleteTopics (API key 20). Issue #3546 asks the gateway to delete the Iggy stream and the Iggy topic. Unmapped Kafka topics share one default stream, and a mapping override can put two Kafka topics in the same stream, so deleting that stream would remove topics the caller did not name. This change deletes only the Iggy topic the Kafka name resolves to, and leaves the stream in place even when the delete leaves it empty. DescribeConfigs and AlterConfigs are not part of this change.
## What changed?
DeleteTopics was outside the scoped API set, so the gateway closed the connection. A reading of #3546 that also deletes the Iggy stream would drop other Kafka topics that still share that stream.
Versions 1 through 5 are now advertised (version 6, the topic-id shape, is not). With the bridge on, each name deletes only the mapped Iggy topic. Success is error code 0. A missing topic returns UNKNOWN_TOPIC_OR_PARTITION (3); on version 5, where `error_message` is encoded, that result's text is the fixed string "internal error deleting this topic". A name that is empty, has leading or trailing whitespace, is longer than 249 bytes, is "." or "..", or contains a character outside letters, digits, ".", "_", and "-" returns INVALID_TOPIC_EXCEPTION and is not sent to Iggy. The validation reason is included as `error_message` on version 5. Versions 1 through 4 encode the error code only.
With the bridge off, the request is still validated and every named topic is answered NOT_CONTROLLER (41), so the client does not treat the delete as done. More than 100 topic names in one request, duplicates included, returns POLICY_VIOLATION (44) for every name and deletes nothing. One missing or invalid name does not stop another name in the same request from being deleted. A name repeated in one request is removed on the first occurrence, and the later occurrence returns UNKNOWN_TOPIC_OR_PARTITION because the topic is already gone.
## Local Execution
- Passed : Yes
- Pre-commit hooks ran : The following prehooks ran
Ran the following tests and are passed :
- `clamp_request_timeout_rejects_a_zero_or_negative_value_up_to_the_floor`
- `clamp_request_timeout_caps_an_oversized_value_at_the_ceiling`
- `clamp_request_timeout_passes_through_a_reasonable_value_unchanged`
- `delete_topics_v5_huge_topic_names_count_rejected`
- `delete_topics_v5_desync_varint_poc_rejected`
- `delete_topics_v1_truncated_legacy_topic_name_rejected`
- `delete_topics_deletes_a_real_iggy_topic`
- `delete_topics_never_deletes_the_backing_stream_even_when_it_becomes_empty`
- `delete_topics_on_a_nonexistent_topic_returns_unknown_topic_or_partition`
- `delete_topics_second_occurrence_of_a_duplicate_name_is_unknown_topic_or_partition`
- `delete_topics_deletes_existing_topics_alongside_a_missing_one`
- `delete_topics_rejects_more_than_the_topic_cap`
- `delete_topics_over_cap_error_message_names_the_limit`
- `given_kafka_topics_sh_when_creating_then_deleting_a_topic_should_round_trip_through_the_bridge` (`kafka-topics.sh` create, list, delete, list)
- `supported_ranges_table_has_thirteen_entries`
- `golden_apiversions_v3_flexible_response_fixture`
- `golden_apiversions_v1_response_fixture`
```bash
cargo test -p iggy-gateway-kafka --lib delete_topics::tests
cargo test -p iggy-gateway-kafka --lib delete_topics_v5_
cargo test -p iggy-gateway-kafka --lib delete_topics_v1_truncated_legacy_topic_name_rejected
cargo test -p iggy-gateway-kafka --test delete_topics_real_bridge_tests
cargo test -p iggy-gateway-kafka --test kafka_client_e2e_tests given_kafka_topics_sh_when_creating_then_deleting_a_topic_should_round_trip_through_the_bridge
cargo test -p iggy-gateway-kafka --test version_firewall_tests supported_ranges_table_has_thirteen_entries
cargo test -p iggy-gateway-kafka --test golden_wire_fixtures_tests golden_apiversions_
```
## AI Usage
1. Which tools? Claude
2. Scope of usage? Implementation, Review
3. How did you verify the generated code works correctly? Code walkthrough and manual testing
4. Can you explain every line of the code if asked? Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179494226
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,30 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same `default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing risk for create), so a
+ /// stream this call happens to empty may still be the live home for a different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with `StreamNameNotFound`/`TopicNameNotFound`
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4205806476
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
Review Comment:
fixed
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,31 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same `default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing risk for create), so a
+ /// stream this call happens to empty may still be the live home for a different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with `StreamIdNotFound` (1009) or
+ /// `TopicIdNotFound` (2010) if the mapped stream or topic doesn't exist, or for
+ /// connectivity/auth failures.
+ pub async fn delete_kafka_topic(&self, kafka_topic: &str) -> Result<(), BridgeError> {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179492139
##########
gateways/kafka/tests/common/scope.rs:
##########
@@ -37,6 +37,7 @@ pub const SCOPED_API_KEYS: &[(i16, &str, i16, i16)] = &[
(14, "SyncGroup", 0, 5),
(18, "ApiVersions", 0, 3),
(19, "CreateTopics", 2, 5),
+ (20, "DeleteTopics", 1, 5),
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4400:
URL: https://github.com/apache/iggy/pull/4400#issuecomment-5982254219
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4400?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 61.07%. Comparing base ([`2d7fddb`](https://app.codecov.io/gh/apache/iggy/commit/2d7fddba5552640518aa7f1372f69af9c77e436a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1a0acc4`](https://app.codecov.io/gh/apache/iggy/commit/1a0acc49b1c370384243df0b697c9989a9c0ad2a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4400 +/- ##
=============================================
- Coverage 87.96% 61.07% -26.90%
+ Complexity 1579 1578 -1
=============================================
Files 1290 1290
Lines 230145 230145
Branches 193480 193482 +2
=============================================
- Hits 202440 140551 -61889
- Misses 22980 84936 +61956
+ Partials 4725 4658 -67
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.66% <ø> (-32.41%)` | :arrow_down: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.72% <ø> (-0.02%)` | :arrow_down: |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.61% <ø> (-0.02%)` | :arrow_down: |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `91.24% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.67% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.57% <ø> (-0.03%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4400/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.26% <ø> (+0.07%)` | :arrow_up: |
[see 525 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4400/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179496428
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use std::time::Duration;
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, handle_versioned_request, is_supported_version,
+ respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+/// Same bounds as `create_topics::clamp_request_timeout` - this request carries the identical
+/// KIP-4 `timeout_ms` field for the identical reason (a client-supplied deadline that becomes
+/// the aggregate bridge-work budget, otherwise unchecked).
+const MIN_REQUEST_TIMEOUT: Duration = Duration::from_millis(1_000);
+const MAX_REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn clamp_request_timeout(timeout_ms: i32) -> Duration {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4205817221
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect against the way
+/// creating it twice does - the first occurrence deletes it, the second then answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline, delete_one_topic(bridge, name)).await {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4400:
URL: https://github.com/apache/iggy/pull/4400#issuecomment-6078400128
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179492684
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use std::time::Duration;
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, handle_versioned_request, is_supported_version,
+ respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+/// Same bounds as `create_topics::clamp_request_timeout` - this request carries the identical
+/// KIP-4 `timeout_ms` field for the identical reason (a client-supplied deadline that becomes
+/// the aggregate bridge-work budget, otherwise unchecked).
+const MIN_REQUEST_TIMEOUT: Duration = Duration::from_millis(1_000);
+const MAX_REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn clamp_request_timeout(timeout_ms: i32) -> Duration {
+ let requested = Duration::from_millis(u64::try_from(timeout_ms).unwrap_or(0));
+ requested.clamp(MIN_REQUEST_TIMEOUT, MAX_REQUEST_TIMEOUT)
+}
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect against the way
+/// creating it twice does - the first occurrence deletes it, the second then answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline, delete_one_topic(bridge, name)).await {
+ Ok(result) => result,
+ Err(_elapsed) => {
+ tracing::warn!(
+ kafka_topic = name.as_str(),
+ "DeleteTopics: this topic's bridge work exceeded the request deadline; \
+ answering retriable instead of blocking further"
+ );
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_REQUEST_TIMED_OUT)
+ .with_error_message(None)
+ }
+ };
+ results.push(result);
+ }
+ results
+}
+
+async fn delete_one_topic(bridge: &IggyBridge, name: &TopicName) -> DeletableTopicResult {
+ let result = DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_message(None);
+ match bridge.delete_kafka_topic(name.as_str()).await {
+ // The second arm: the write committed on its first attempt, the SDK's own reconnect path
+ // replayed it, and the server's client-table dedup caught the replay - not a fault. Same
+ // reasoning as `create_topics::create_one_topic`'s identical arm.
+ Ok(()) | Err(BridgeError::Iggy(IggyError::RequestAlreadyApplied)) => {
+ result.with_error_code(ERROR_NONE)
+ }
+ Err(err) => bridge_error_result(result, name.as_str(), &err),
+ }
+}
+
+/// Maps a bridge failure to a Kafka result, logging the real cause server-side.
+///
+/// Same caution as `create_topics::bridge_error_result` on forwarding `err.to_string()`: a
+/// server-side rejection can reconstruct with default-valued fields, so only the client-caused
+/// variant (an invalid name) forwards its own text; everything else gets a fixed message.
+fn bridge_error_result(
+ result: DeletableTopicResult,
+ kafka_topic: &str,
+ err: &BridgeError,
+) -> DeletableTopicResult {
+ let error_code = err.to_kafka_error_code();
+ match err {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(
+ kafka_topic,
+ reason,
+ "DeleteTopics rejected an invalid topic name"
+ );
+ result
+ .with_error_code(error_code)
+ .with_error_message(Some(StrBytes::from(reason.clone())))
+ }
+ other => {
+ tracing::error!(kafka_topic, %other, "DeleteTopics failed against the Iggy bridge");
+ result
+ .with_error_code(error_code)
+ .with_error_message(Some(StrBytes::from(
+ "internal error deleting this topic".to_string(),
Review Comment:
fixed
##########
gateways/kafka/docs/SCOPE.md:
##########
@@ -227,6 +229,29 @@ below it are still open for the issues that build on top of it.
request's own wire `timeout_ms` (clamped to `[1s, 30s]`) now bounds the whole handler's
aggregate bridge work, not just decoded and discarded - a deadline that fires answers every
topic `REQUEST_TIMED_OUT` rather than continuing to hold the shared lockstep `IggyClient`.
+- [x] Real DeleteTopics ([#3546](https://github.com/apache/iggy/issues/3546)): with
+ `IGGY_KAFKA_BRIDGE_ENABLED=true`, deletes the Iggy topic through
+ `IggyBridge::delete_kafka_topic`. `src/protocol/handlers/delete_topics.rs`,
+ `tests/delete_topics_real_bridge_tests.rs`. With the bridge off, the stub answers
+ `NOT_CONTROLLER` (41), matching CreateTopics' own stub convention.
+ - **Never deletes the backing Iggy stream**, even when the delete leaves it with no topics in
+ it. `TopicMapping::resolve` sends every unmapped Kafka topic to the same `default_stream`
+ (`ensure_stream_and_topic`'s own doc comment flags the identical sharing risk for create), so
+ a stream this call happens to empty may still be the live home for a different Kafka topic a
+ moment later - there is no way to tell the two cases apart from here, and real Kafka itself
+ has no analogous container object for a `DeleteTopics` to clean up. See
+ [`BRIDGE_MAPPING.md`](BRIDGE_MAPPING.md).
+ - Per-topic results are independent: one missing or invalid name in a batch does not block an
+ existing, validly-named topic elsewhere in the same request from being deleted.
+ - A name repeated within one request is not specially rejected the way `CreateTopics` rejects a
+ duplicate - deleting the same topic twice has no race to protect against the way creating it
+ twice does, so the first occurrence deletes it and the second then correctly answers
+ `UNKNOWN_TOPIC_OR_PARTITION` (3) because the topic genuinely no longer exists.
+ - Same bridge-fan-out bounds as CreateTopics: a request naming more than 100 distinct topics is
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4205808389
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
Review Comment:
fixed
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,31 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same `default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing risk for create), so a
+ /// stream this call happens to empty may still be the live home for a different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with `StreamIdNotFound` (1009) or
+ /// `TopicIdNotFound` (2010) if the mapped stream or topic doesn't exist, or for
+ /// connectivity/auth failures.
+ pub async fn delete_kafka_topic(&self, kafka_topic: &str) -> Result<(), BridgeError> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect against the way
+/// creating it twice does - the first occurrence deletes it, the second then answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline, delete_one_topic(bridge, name)).await {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4400:
URL: https://github.com/apache/iggy/pull/4400#issuecomment-6050340120
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179121121
##########
gateways/kafka/tests/common/scope.rs:
##########
@@ -37,6 +37,7 @@ pub const SCOPED_API_KEYS: &[(i16, &str, i16, i16)] = &[
(14, "SyncGroup", 0, 5),
(18, "ApiVersions", 0, 3),
(19, "CreateTopics", 2, 5),
+ (20, "DeleteTopics", 1, 5),
Review Comment:
warning: The change puts key 20 in `SCOPED_API_KEYS`, but `OUT_OF_SCOPE_API_KEYS` still lists it at `tests/common/wire.rs:39`, so the two tables contradict each other. `out_of_scope_api_keys_close_without_panic` now passes because the v0 response fails to encode, not because the key is unknown, and `server_e2e_tests.rs:425` hides the contradiction with `[..4]`. Remove the `(20, "DeleteTopics")` entry and that slice.
##########
gateways/kafka/tests/version_firewall_tests.rs:
##########
@@ -100,9 +101,10 @@ fn is_supported_version_matches_scope_table() {
/// `advertised_min_version` - the same reasoning applies to it as the function under test.
///
/// Relies on `SUPPORTED_RANGES` (src) and `SCOPED_API_KEYS` (test) sharing declaration order
-/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`, `FindCoordinator`,
-/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) - `supported_ranges_table_has_twelve_entries` plus
-/// `is_supported_version_matches_scope_table` already pin that both tables cover the same keys.
+/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`, `DeleteTopics`,
Review Comment:
nit: The comment says the test relies on the two tables sharing declaration order, but `api_versions.rs:86` sorts the advertisement by key and the declarations differ. Say that the advertisement is key-ascending, and add `InitProducerId` to the listed names.
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use std::time::Duration;
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, handle_versioned_request, is_supported_version,
+ respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+/// Same bounds as `create_topics::clamp_request_timeout` - this request carries the identical
+/// KIP-4 `timeout_ms` field for the identical reason (a client-supplied deadline that becomes
+/// the aggregate bridge-work budget, otherwise unchecked).
+const MIN_REQUEST_TIMEOUT: Duration = Duration::from_millis(1_000);
+const MAX_REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn clamp_request_timeout(timeout_ms: i32) -> Duration {
Review Comment:
simplification: `MIN_REQUEST_TIMEOUT`, `MAX_REQUEST_TIMEOUT`, `clamp_request_timeout`, and the three tests at line 270 copy `create_topics.rs` verbatim. Simpler: move one copy into `handlers/mod.rs` as `pub(crate)` and call it from both handlers. Saves about 20 lines and keeps one place for the bounds.
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use std::time::Duration;
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, handle_versioned_request, is_supported_version,
+ respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+/// Same bounds as `create_topics::clamp_request_timeout` - this request carries the identical
+/// KIP-4 `timeout_ms` field for the identical reason (a client-supplied deadline that becomes
+/// the aggregate bridge-work budget, otherwise unchecked).
+const MIN_REQUEST_TIMEOUT: Duration = Duration::from_millis(1_000);
+const MAX_REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn clamp_request_timeout(timeout_ms: i32) -> Duration {
+ let requested = Duration::from_millis(u64::try_from(timeout_ms).unwrap_or(0));
+ requested.clamp(MIN_REQUEST_TIMEOUT, MAX_REQUEST_TIMEOUT)
+}
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect against the way
+/// creating it twice does - the first occurrence deletes it, the second then answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline, delete_one_topic(bridge, name)).await {
+ Ok(result) => result,
+ Err(_elapsed) => {
+ tracing::warn!(
+ kafka_topic = name.as_str(),
+ "DeleteTopics: this topic's bridge work exceeded the request deadline; \
+ answering retriable instead of blocking further"
+ );
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_REQUEST_TIMED_OUT)
+ .with_error_message(None)
+ }
+ };
+ results.push(result);
+ }
+ results
+}
+
+async fn delete_one_topic(bridge: &IggyBridge, name: &TopicName) -> DeletableTopicResult {
+ let result = DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_message(None);
+ match bridge.delete_kafka_topic(name.as_str()).await {
+ // The second arm: the write committed on its first attempt, the SDK's own reconnect path
+ // replayed it, and the server's client-table dedup caught the replay - not a fault. Same
+ // reasoning as `create_topics::create_one_topic`'s identical arm.
+ Ok(()) | Err(BridgeError::Iggy(IggyError::RequestAlreadyApplied)) => {
+ result.with_error_code(ERROR_NONE)
+ }
+ Err(err) => bridge_error_result(result, name.as_str(), &err),
+ }
+}
+
+/// Maps a bridge failure to a Kafka result, logging the real cause server-side.
+///
+/// Same caution as `create_topics::bridge_error_result` on forwarding `err.to_string()`: a
+/// server-side rejection can reconstruct with default-valued fields, so only the client-caused
+/// variant (an invalid name) forwards its own text; everything else gets a fixed message.
+fn bridge_error_result(
+ result: DeletableTopicResult,
+ kafka_topic: &str,
+ err: &BridgeError,
+) -> DeletableTopicResult {
+ let error_code = err.to_kafka_error_code();
+ match err {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(
+ kafka_topic,
+ reason,
+ "DeleteTopics rejected an invalid topic name"
+ );
+ result
+ .with_error_code(error_code)
+ .with_error_message(Some(StrBytes::from(reason.clone())))
+ }
+ other => {
+ tracing::error!(kafka_topic, %other, "DeleteTopics failed against the Iggy bridge");
+ result
+ .with_error_code(error_code)
+ .with_error_message(Some(StrBytes::from(
+ "internal error deleting this topic".to_string(),
Review Comment:
warning: A missing topic and an `Unauthorized` reply both land in this catch-all. A v5 client then reads `internal error deleting this topic` for a topic that does not exist, and the log records that routine case at `error!`. Add arms for the not-found and authorization variants that answer accurate text and log at `debug!`.
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,30 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same `default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing risk for create), so a
+ /// stream this call happens to empty may still be the live home for a different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with `StreamNameNotFound`/`TopicNameNotFound`
Review Comment:
nit: The doc names `StreamNameNotFound` and `TopicNameNotFound`, but a rejection from `delete_topic` arrives as `StreamIdNotFound` (1009) or `TopicIdNotFound` (2010). Name those two variants instead.
##########
gateways/kafka/docs/SCOPE.md:
##########
@@ -227,6 +229,29 @@ below it are still open for the issues that build on top of it.
request's own wire `timeout_ms` (clamped to `[1s, 30s]`) now bounds the whole handler's
aggregate bridge work, not just decoded and discarded - a deadline that fires answers every
topic `REQUEST_TIMED_OUT` rather than continuing to hold the shared lockstep `IggyClient`.
+- [x] Real DeleteTopics ([#3546](https://github.com/apache/iggy/issues/3546)): with
+ `IGGY_KAFKA_BRIDGE_ENABLED=true`, deletes the Iggy topic through
+ `IggyBridge::delete_kafka_topic`. `src/protocol/handlers/delete_topics.rs`,
+ `tests/delete_topics_real_bridge_tests.rs`. With the bridge off, the stub answers
+ `NOT_CONTROLLER` (41), matching CreateTopics' own stub convention.
+ - **Never deletes the backing Iggy stream**, even when the delete leaves it with no topics in
+ it. `TopicMapping::resolve` sends every unmapped Kafka topic to the same `default_stream`
+ (`ensure_stream_and_topic`'s own doc comment flags the identical sharing risk for create), so
+ a stream this call happens to empty may still be the live home for a different Kafka topic a
+ moment later - there is no way to tell the two cases apart from here, and real Kafka itself
+ has no analogous container object for a `DeleteTopics` to clean up. See
+ [`BRIDGE_MAPPING.md`](BRIDGE_MAPPING.md).
+ - Per-topic results are independent: one missing or invalid name in a batch does not block an
+ existing, validly-named topic elsewhere in the same request from being deleted.
+ - A name repeated within one request is not specially rejected the way `CreateTopics` rejects a
+ duplicate - deleting the same topic twice has no race to protect against the way creating it
+ twice does, so the first occurrence deletes it and the second then correctly answers
+ `UNKNOWN_TOPIC_OR_PARTITION` (3) because the topic genuinely no longer exists.
+ - Same bridge-fan-out bounds as CreateTopics: a request naming more than 100 distinct topics is
Review Comment:
nit: This bullet says a request naming more than 100 distinct topics is rejected, but the handler at `delete_topics.rs:121` counts duplicates too. Say `more than 100 topic names`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4400:
URL: https://github.com/apache/iggy/pull/4400#issuecomment-5983796825
/skill team-review-slim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4194767994
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,31 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same `default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing risk for create), so a
+ /// stream this call happens to empty may still be the live home for a different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with `StreamIdNotFound` (1009) or
+ /// `TopicIdNotFound` (2010) if the mapped stream or topic doesn't exist, or for
+ /// connectivity/auth failures.
+ pub async fn delete_kafka_topic(&self, kafka_topic: &str) -> Result<(), BridgeError> {
Review Comment:
this irreversible delete runs as the bridge's Iggy user and the Kafka principal is discarded, so with SASL on any authenticated principal can delete any topic in the default/override streams, even one DescribeAcls shows without DELETE. The README warning only covers SASL off, which implies SASL on protects it.
With SASL on, gate on `principal.permissions.manage_topics` (auth.rs:512), answer 29, and fail closed when permissions aren't known. A kill switch answering 73 `TOPIC_DELETION_DISABLED` would be worth having too.
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect against the way
+/// creating it twice does - the first occurrence deletes it, the second then answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline, delete_one_topic(bridge, name)).await {
Review Comment:
`timeout_at` polls the inner future before it checks the delay, and that first poll already reaches the SDK's spawned write (core/sdk/src/tcp/tcp_client.rs:1517). So every name past the deadline still sends a real delete while the client is told `REQUEST_TIMED_OUT`. A late read also drops the shared bridge stream (tcp_client.rs:1655), so every Kafka connection's next Produce/Metadata gets `NotConnected` and a reconnect.
Check `Instant::now() >= deadline` before dispatching and answer 7 with no call, like list_offsets.rs:196, and add a unit test. The comments at :139 and :161 and SCOPE.md:252 need the same fix, and create_topics has the same shape.
##########
gateways/kafka/src/protocol/bounds_guard.rs:
##########
@@ -703,6 +703,44 @@ pub fn validate_create_topics_shape(
Ok(())
}
+/// Mirrors the field order `DeleteTopicsRequest::decode` walks, for the `topic_names`-based
+/// shape (v1-v5) only.
+///
+/// This bridge's `RANGE` never advertises v6, the `topics: Vec<DeleteTopicState>`
+/// (topic-id-based) shape, so that branch is never reached and isn't walked here.
+///
+/// # Errors
+///
+/// Returns an error when a declared array/string/bytes length cannot fit in the bytes remaining
+/// in the frame, or the body is truncated or malformed in a way that cannot be walked.
+pub fn validate_delete_topics_shape(
+ version: i16,
+ body: &Bytes,
+ max_frame_size: usize,
+) -> Result<()> {
+ let mut c = ShapeCursor::new(body.clone(), max_frame_size);
+ let flexible = version >= 4;
Review Comment:
nothing pins v1-v4. Changing this to `version >= 5` survives all 548 no-server tests and would reject every v4 request (Java 2.4-2.6). Changing the bridge-off stub at delete_topics.rs:265 from 41 to NONE, which makes a client believe the delete happened, also survives. version_firewall_tests.rs:498 sends only an empty request.
Add a no-server handler test with one name at v1/v3/v4/v5 that asserts a per-topic 41.
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
Review Comment:
KRaft `ControllerApis.deleteTopics` answers 42 "Duplicate topic name." for every occurrence and deletes nothing. Here the first occurrence deletes the topic, so the divergence goes the destructive way, and this crate's own CreateTopics rejects duplicates (`find_duplicate_names`).
Pre-scan the request and answer 42 for each occurrence with no bridge call, then update the real-bridge test (:204-228) and SCOPE.md:246.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4205811541
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why: the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no way to tell
+//! whether the stream it just emptied is actually abandoned or just temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+ (`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded at every version it
+//! supports.
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse, TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT, GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ clamp_request_timeout, decode_guarded, encode_message, handle_versioned_request,
+ is_supported_version, respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`, topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name `TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted differently: every
+/// name here costs its own bridge round trip against the single lockstep `IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) -> HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS, api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty - topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually (`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single `timeout` around the
+/// entire call would discard every already-resolved result the moment one topic's call ran
+/// long, answering a retriable code even for topics that had already deleted cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way `CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect against the way
+/// creating it twice does - the first occurrence deletes it, the second then answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline, delete_one_topic(bridge, name)).await {
Review Comment:
fixed
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,31 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same `default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing risk for create), so a
+ /// stream this call happens to empty may still be the live home for a different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with `StreamIdNotFound` (1009) or
+ /// `TopicIdNotFound` (2010) if the mapped stream or topic doesn't exist, or for
+ /// connectivity/auth failures.
+ pub async fn delete_kafka_topic(&self, kafka_topic: &str) -> Result<(), BridgeError> {
Review Comment:
fixed
##########
gateways/kafka/src/protocol/bounds_guard.rs:
##########
@@ -703,6 +703,44 @@ pub fn validate_create_topics_shape(
Ok(())
}
+/// Mirrors the field order `DeleteTopicsRequest::decode` walks, for the `topic_names`-based
+/// shape (v1-v5) only.
+///
+/// This bridge's `RANGE` never advertises v6, the `topics: Vec<DeleteTopicState>`
+/// (topic-id-based) shape, so that branch is never reached and isn't walked here.
+///
+/// # Errors
+///
+/// Returns an error when a declared array/string/bytes length cannot fit in the bytes remaining
+/// in the frame, or the body is truncated or malformed in a way that cannot be walked.
+pub fn validate_delete_topics_shape(
+ version: i16,
+ body: &Bytes,
+ max_frame_size: usize,
+) -> Result<()> {
+ let mut c = ShapeCursor::new(body.clone(), max_frame_size);
+ let flexible = version >= 4;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4400:
URL: https://github.com/apache/iggy/pull/4400#issuecomment-6028530477
/author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4205804373
##########
gateways/kafka/src/protocol/bounds_guard.rs:
##########
@@ -703,6 +703,44 @@ pub fn validate_create_topics_shape(
Ok(())
}
+/// Mirrors the field order `DeleteTopicsRequest::decode` walks, for the `topic_names`-based
+/// shape (v1-v5) only.
+///
+/// This bridge's `RANGE` never advertises v6, the `topics: Vec<DeleteTopicState>`
+/// (topic-id-based) shape, so that branch is never reached and isn't walked here.
+///
+/// # Errors
+///
+/// Returns an error when a declared array/string/bytes length cannot fit in the bytes remaining
+/// in the frame, or the body is truncated or malformed in a way that cannot be walked.
+pub fn validate_delete_topics_shape(
+ version: i16,
+ body: &Bytes,
+ max_frame_size: usize,
+) -> Result<()> {
+ let mut c = ShapeCursor::new(body.clone(), max_frame_size);
+ let flexible = version >= 4;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179495638
##########
gateways/kafka/tests/version_firewall_tests.rs:
##########
@@ -100,9 +101,10 @@ fn is_supported_version_matches_scope_table() {
/// `advertised_min_version` - the same reasoning applies to it as the function under test.
///
/// Relies on `SUPPORTED_RANGES` (src) and `SCOPED_API_KEYS` (test) sharing declaration order
-/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`, `FindCoordinator`,
-/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) - `supported_ranges_table_has_twelve_entries` plus
-/// `is_supported_version_matches_scope_table` already pin that both tables cover the same keys.
+/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`, `DeleteTopics`,
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4400:
URL: https://github.com/apache/iggy/pull/4400#issuecomment-5984850573
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(gateways): Kafka Gateway add DeleteTopics API Key that deletes the Iggy topic only [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio merged PR #4400:
URL: https://github.com/apache/iggy/pull/4400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org