You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iggy.apache.org by "ryerraguntla (via GitHub)" <gi...@apache.org> on 2026/05/14 00:58:23 UTC
feature - Technical implementation discussion of kafka to iggy bridge
GitHub user ryerraguntla created a discussion: feature - Technical implementation discussion of kafka to iggy bridge
The discussion is branching out the discussion from https://github.com/apache/iggy/discussions/3081 around the architectural and design of the kafka_iggy_bridge . The design details will be added here for community discussion and sharing the implementation idea. All the implementation jira tickets will be added here.
GitHub link: https://github.com/apache/iggy/discussions/3252
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla edited a comment on the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Below is the proposed architecture for the version 0.01 of kafka_2_iggy bridge. My thought process is to provide a bridge for transition to iggy with the current iggy capabilities. As Iggy's new server and SDK is available, the known gaps by design will be taken care.
### **Architecture Overview**
**Approach:** **Standalone TCP Proxy** (referenced as Option 1 in discussion #3081, comment #3044)
The bridge runs as a separate process that:
1. **Listens** on port 9093 (Kafka wire protocol)
2. **Decodes** Kafka binary requests (ApiVersions, Produce, Fetch, etc.)
3. **Translates** to Iggy SDK calls via `IggyBridge`
4. **Encodes** Kafka-compliant responses and sends them back to the client
### **Key Technical Achievements**
#### **1. Complete Protocol Foundation**
**Files:** `src/protocol/codec.rs`, `src/protocol/header.rs`
Implement a zero-copy encoder/decoder for all Kafka primitives:
---
#### **2. Request/Response Header Auto-detection**
**File:** `src/protocol/header.rs`
Kafka has **two incompatible header formats**:
- **v1 (non-flexible):** `api_key | api_version | correlation_id | client_id (NULLABLE_STRING)`
- **v2 (flexible):** `api_key | api_version | correlation_id | client_id (COMPACT_NULLABLE_STRING) | tagged_fields`
The bridge **auto-detects** which format to use based on a lookup table covering all 65 API keys:
**Impact:** Clients using old versions (e.g., Kafka 2.x) and new versions (Kafka 3.x+) work seamlessly against the same bridge.
---
#### **3. Kafka-to-Iggy Mapping Layer**
**File:** `src/iggy_bridge.rs`
The `IggyBridge` struct wraps the Iggy SDK and provides **idempotent** operations:
| Kafka Concept | Iggy Mapping | Method |
|---------------|--------------|--------|
| **Topic** | Stream + Topic (same name) | `ensure_stream_and_topic()` |
| **Partition** | Balanced partitioning (Iggy decides) | `produce()` |
| **Offset** | Message offset (0-based, same as Kafka) | `fetch()` |
| **High watermark** | Current end offset | `high_watermark()` |
**Key Design Decision:** Kafka partition IDs are **ignored** during produce. The bridge uses `Partitioning::balanced()` so Iggy handles partition assignment. This avoids 0-vs-1-based index confusion and leverages Iggy's native load balancing. This implementation will be revisited after the iggy partition mapping is implemented. For now it will be same partition id.
---
#### **4. Implemented API Keys (Phase 1)**
| API Key | Name | Versions | Iggy Integration |
|---------|------|----------|--------|------------------|
| **18** | ApiVersions | v0–v3 | N/A (handshake only) |
| **3** | Metadata | v0–v9 |Returns stub (single broker at 127.0.0.1:9093) |
| **0** | Produce | v3–v9 | `send_messages()` with balanced partitioning |
| **1** | Fetch | v4–v12 | `poll_messages()` with offset-based polling |
| **2** | ListOffsets | v1–v6 | `high_watermark()` for latest offset |
| **19** | CreateTopics | v2–v5 | `create_stream()` + `create_topic()` |
**Code Locations:**
- Decoders: `src/protocol/requests.rs`
- Encoders: `src/protocol/responses.rs`
- Iggy integration: `src/protocol/api.rs` (`handle_request_with_iggy()`)
---
### **5. Comprehensive Testing Infrastructure**
#### **Unit Tests**
- **Codec tests** (`tests/codec_tests.rs`): Varint encoding, nullable strings, compact arrays
- **Header tests** (`tests/header_tests.rs`): Header v1/v2 round-trip, correlation ID preservation
- **API handler tests** (`tests/api_handler_tests.rs`): ApiVersions flexible/legacy encoding
#### **Integration Tests**
- **Server lifecycle** (`tests/server_integration_tests.rs`): Connection handling, frame size limits, graceful shutdown
- **Golden wire fixtures** (`tests/golden_wire_fixtures_tests.rs`): Byte-exact validation against known-good Kafka messages
#### **End-to-End Testing Tool**
**`tools/kafka-tool`**: A CLI that generates **280+ valid Kafka wire messages** (every API key × every version) using the `kafka-protocol` crate.
**Purpose:** Ensures the bridge doesn't crash or produce malformed responses when faced with any valid Kafka request.
---
### **Technical Challenges Solved**
#### **Challenge 1: Flexible vs. Legacy Encoding**
Kafka introduced "flexible" encoding (compact arrays, tagged fields) gradually across versions. For example:
- **Produce v9+** is flexible
- **Produce v3–v8** is legacy
- **ApiVersions v3+** is flexible
- **Metadata v9+** is flexible (but v0–v8 is legacy)
**Solution:** Version-aware encoder/decoder that checks `api_version >= flexible_threshold` and calls the appropriate codec methods (`write_compact_nullable_string()` vs. `write_nullable_string()`).
---
#### **Challenge 2: RecordBatch Format**
Kafka Produce requests contain a **RecordBatch** (Kafka's internal message format) with:
- Compression (gzip, snappy, lz4, zstd)
- Timestamps (millisecond precision)
- Headers (key-value metadata)
- Checksums (CRC32C)
**Current Solution:** The bridge treats RecordBatch as **opaque bytes** and stores them directly in Iggy. This preserves the original format but means:
- Iggy can't filter on Kafka message headers
- Iggy can't decompress Kafka messages for storage optimization
- Kafka consumers can decode the RecordBatch natively
**Future Enhancement:** Parse RecordBatch and extract individual messages, storing them as Iggy messages with metadata preserved in headers.
---
#### **Challenge 3: Partition Mapping**
**Problem:** Kafka partitions are 0-based (0, 1, 2, ...), while Iggy partitions are 1-based (1, 2, 3, ...).
**Solution:** The bridge **ignores** Kafka partition IDs during produce and uses `Partitioning::balanced()`. For fetch, it polls across all partitions and returns messages in offset order. This avoids off-by-one bugs and leverages Iggy's native balancing.
**Trade-off:** Kafka clients lose control over which partition receives a message. For sticky-key partitioning (e.g., all messages for `user_id=123` go to partition 0), a future enhancement would be:
TODO:
Design for this will be revisited after the server-ng and in works features are available.
---
### **Comparison to "Built-in Listener" Approach**
Discussion #3081 mentions integrating Kafka protocol directly into Iggy server. Here's why the **standalone proxy is superior**:
| Concern | Standalone Proxy | Built-in Listener |
|---------|------------------|-------------------|
| **Iggy codebase impact** | Zero (pure client) | High (new TCP listener + dispatcher) |
| **Testing complexity** | Isolated unit tests | Requires Iggy server in test environment |
| **Deployment flexibility** | Optional (run only if Kafka compat needed) | Always-on (port 9093 always open) |
| **Version evolution** | Independent releases | Tied to Iggy release cycle |
| **Security boundary** | Proxy can enforce Kafka ACLs separately | ACLs mixed with Iggy auth |
| **Latency** | +1-2ms | ~0ms |
**Verdict:** Standalone proxy is the **pragmatic choice** for MVP. Built-in listener could be considered for v2 if latency becomes a bottleneck (unlikely for most workloads).
---
### **Next Steps (Phase 2 API Keys)**
To achieve **full producer/consumer compatibility**, the following API keys are required:
#### **Critical (Producer)**
- **Produce**
- **ApiVersions**
- **Metadata**
- **InitProducerId** (API key 22) — needed for idempotent/transactional producers
- **AddPartitionsToTxn** (API key 24) — transactional writes
- **EndTxn** (API key 26) — commit/abort transactions
#### **Critical (Consumer)**
- **Fetch**
- **ListOffsets**
- **FindCoordinator** (API key 10) — locate consumer group coordinator
- **JoinGroup** (API key 11) — join consumer group
- **SyncGroup** (API key 14) — receive partition assignment
- **Heartbeat** (API key 12) — keep group membership alive
- **OffsetCommit** (API key 8) — commit consumer offsets
- **OffsetFetch** (API key 9) — fetch last committed offsets
- **LeaveGroup** (API key 13) — graceful consumer shutdown
---
### **Known Limitations**
1. **Consumer Groups:** Not implemented. Each Kafka consumer currently acts as an independent Iggy consumer. No rebalancing or offset tracking.
2. **Transactions:** Not supported. `InitProducerId`, `AddPartitionsToTxn`, and `EndTxn` return `UNSUPPORTED_VERSION`.
3. **Compression:** RecordBatch compression is preserved (opaque bytes) but not decompressed. Iggy stores the compressed payload as-is.
4. **ACLs/Auth:** The bridge authenticates to Iggy as root. Kafka client auth (SASL, mTLS) is not mapped to Iggy users.
5. **Partition Assignment:** Kafka clients cannot control which partition receives a message (balanced partitioning only).
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-16911781
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla edited a comment on the discussion: feature - Technical implementation discussion of kafka to iggy bridge
I concur with @kparisa from the point of hosting services. Being an open source migration, offer a bridge to taste the benefits of Iggy and save hosting costs with SDKs with commercial support. Professional services can be offered on need basis (on a lighter note creating job opportunities in the current job market). Near term it will be a quick win with a bridge then migration with SDKs. if @spetz approves the proposed technical approach then I can work on the implementation phases with other community members.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17152946
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
@ryerraguntla sounds good, feel free to start working on it. I'm not sure yet if we would add this directly to this repository, or create a new one purely for such a proxy (and future SDKs), but that's just a matter of code organization.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17153596
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
With Option 2, we can start with 1 or 2 main SDKs (Rust, Python/Java), other SDKs can catch up if anyone wants to migrate from Kafka to Iggy easily. Most prominent use case would be migrating from Kafka to Iggy to take advantage of ultra low latencies and significant system resource efficiencies.
My worry with Option 1: Proxy (though it might be safest) is extra infrastructure that needs to be provisioned and scaled!
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17130619
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Yes, starting with the proxy is fine. As Iggy evolves, we can also explore other approaches, including SDK-level implementations.
@ryerraguntla thanks for jumping on this.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17156193
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla edited a comment on the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Below is the proposed architecture for the version 0.01 of kafka_2_iggy bridge. My thought process is provide a bridge for transition to iggy with the current iggy capabilities. As Iggy's new server and SDK is available, the known gaps by design will be taken care.
### **Architecture Overview**
**Approach:** **Standalone TCP Proxy** (referenced as Option 1 in discussion #3081, comment #3044)
The bridge runs as a separate process that:
1. **Listens** on port 9093 (Kafka wire protocol)
2. **Decodes** Kafka binary requests (ApiVersions, Produce, Fetch, etc.)
3. **Translates** to Iggy SDK calls via `IggyBridge`
4. **Encodes** Kafka-compliant responses and sends them back to the client
### **Key Technical Achievements**
#### **1. Complete Protocol Foundation**
**Files:** `src/protocol/codec.rs`, `src/protocol/header.rs`
Implement a zero-copy encoder/decoder for all Kafka primitives:
---
#### **2. Request/Response Header Auto-detection**
**File:** `src/protocol/header.rs`
Kafka has **two incompatible header formats**:
- **v1 (non-flexible):** `api_key | api_version | correlation_id | client_id (NULLABLE_STRING)`
- **v2 (flexible):** `api_key | api_version | correlation_id | client_id (COMPACT_NULLABLE_STRING) | tagged_fields`
The bridge **auto-detects** which format to use based on a lookup table covering all 65 API keys:
**Impact:** Clients using old versions (e.g., Kafka 2.x) and new versions (Kafka 3.x+) work seamlessly against the same bridge.
---
#### **3. Kafka-to-Iggy Mapping Layer**
**File:** `src/iggy_bridge.rs`
The `IggyBridge` struct wraps the Iggy SDK and provides **idempotent** operations:
| Kafka Concept | Iggy Mapping | Method |
|---------------|--------------|--------|
| **Topic** | Stream + Topic (same name) | `ensure_stream_and_topic()` |
| **Partition** | Balanced partitioning (Iggy decides) | `produce()` |
| **Offset** | Message offset (0-based, same as Kafka) | `fetch()` |
| **High watermark** | Current end offset | `high_watermark()` |
**Key Design Decision:** Kafka partition IDs are **ignored** during produce. The bridge uses `Partitioning::balanced()` so Iggy handles partition assignment. This avoids 0-vs-1-based index confusion and leverages Iggy's native load balancing. This implementation will be revisited after the iggy partition mapping is implemented. For now it will be same partition id.
---
#### **4. Implemented API Keys (Phase 1)**
| API Key | Name | Versions | Iggy Integration |
|---------|------|----------|--------|------------------|
| **18** | ApiVersions | v0–v3 | N/A (handshake only) |
| **3** | Metadata | v0–v9 |Returns stub (single broker at 127.0.0.1:9093) |
| **0** | Produce | v3–v9 | `send_messages()` with balanced partitioning |
| **1** | Fetch | v4–v12 | `poll_messages()` with offset-based polling |
| **2** | ListOffsets | v1–v6 | `high_watermark()` for latest offset |
| **19** | CreateTopics | v2–v5 | `create_stream()` + `create_topic()` |
**Code Locations:**
- Decoders: `src/protocol/requests.rs`
- Encoders: `src/protocol/responses.rs`
- Iggy integration: `src/protocol/api.rs` (`handle_request_with_iggy()`)
---
### **5. Comprehensive Testing Infrastructure**
#### **Unit Tests**
- **Codec tests** (`tests/codec_tests.rs`): Varint encoding, nullable strings, compact arrays
- **Header tests** (`tests/header_tests.rs`): Header v1/v2 round-trip, correlation ID preservation
- **API handler tests** (`tests/api_handler_tests.rs`): ApiVersions flexible/legacy encoding
#### **Integration Tests**
- **Server lifecycle** (`tests/server_integration_tests.rs`): Connection handling, frame size limits, graceful shutdown
- **Golden wire fixtures** (`tests/golden_wire_fixtures_tests.rs`): Byte-exact validation against known-good Kafka messages
#### **End-to-End Testing Tool**
**`tools/kafka-tool`**: A CLI that generates **280+ valid Kafka wire messages** (every API key × every version) using the `kafka-protocol` crate.
**Purpose:** Ensures the bridge doesn't crash or produce malformed responses when faced with any valid Kafka request.
---
### **Technical Challenges Solved**
#### **Challenge 1: Flexible vs. Legacy Encoding**
Kafka introduced "flexible" encoding (compact arrays, tagged fields) gradually across versions. For example:
- **Produce v9+** is flexible
- **Produce v3–v8** is legacy
- **ApiVersions v3+** is flexible
- **Metadata v9+** is flexible (but v0–v8 is legacy)
**Solution:** Version-aware encoder/decoder that checks `api_version >= flexible_threshold` and calls the appropriate codec methods (`write_compact_nullable_string()` vs. `write_nullable_string()`).
---
#### **Challenge 2: RecordBatch Format**
Kafka Produce requests contain a **RecordBatch** (Kafka's internal message format) with:
- Compression (gzip, snappy, lz4, zstd)
- Timestamps (millisecond precision)
- Headers (key-value metadata)
- Checksums (CRC32C)
**Current Solution:** The bridge treats RecordBatch as **opaque bytes** and stores them directly in Iggy. This preserves the original format but means:
- Iggy can't filter on Kafka message headers
- Iggy can't decompress Kafka messages for storage optimization
- Kafka consumers can decode the RecordBatch natively
**Future Enhancement:** Parse RecordBatch and extract individual messages, storing them as Iggy messages with metadata preserved in headers.
---
#### **Challenge 3: Partition Mapping**
**Problem:** Kafka partitions are 0-based (0, 1, 2, ...), while Iggy partitions are 1-based (1, 2, 3, ...).
**Solution:** The bridge **ignores** Kafka partition IDs during produce and uses `Partitioning::balanced()`. For fetch, it polls across all partitions and returns messages in offset order. This avoids off-by-one bugs and leverages Iggy's native balancing.
**Trade-off:** Kafka clients lose control over which partition receives a message. For sticky-key partitioning (e.g., all messages for `user_id=123` go to partition 0), a future enhancement would be:
TODO:
Design for this will be revisited after the server-ng and in works features are available.
---
### **Comparison to "Built-in Listener" Approach**
Discussion #3081 mentions integrating Kafka protocol directly into Iggy server. Here's why the **standalone proxy is superior**:
| Concern | Standalone Proxy | Built-in Listener |
|---------|------------------|-------------------|
| **Iggy codebase impact** | Zero (pure client) | High (new TCP listener + dispatcher) |
| **Testing complexity** | Isolated unit tests | Requires Iggy server in test environment |
| **Deployment flexibility** | Optional (run only if Kafka compat needed) | Always-on (port 9093 always open) |
| **Version evolution** | Independent releases | Tied to Iggy release cycle |
| **Security boundary** | Proxy can enforce Kafka ACLs separately | ACLs mixed with Iggy auth |
| **Latency** | +1-2ms | ~0ms |
**Verdict:** Standalone proxy is the **pragmatic choice** for MVP. Built-in listener could be considered for v2 if latency becomes a bottleneck (unlikely for most workloads).
---
### **Next Steps (Phase 2 API Keys)**
To achieve **full producer/consumer compatibility**, the following API keys are required:
#### **Critical (Producer)**
- **Produce**
- **ApiVersions**
- **Metadata**
- **InitProducerId** (API key 22) — needed for idempotent/transactional producers
- **AddPartitionsToTxn** (API key 24) — transactional writes
- **EndTxn** (API key 26) — commit/abort transactions
#### **Critical (Consumer)**
- **Fetch**
- **ListOffsets**
- **FindCoordinator** (API key 10) — locate consumer group coordinator
- **JoinGroup** (API key 11) — join consumer group
- **SyncGroup** (API key 14) — receive partition assignment
- **Heartbeat** (API key 12) — keep group membership alive
- **OffsetCommit** (API key 8) — commit consumer offsets
- **OffsetFetch** (API key 9) — fetch last committed offsets
- **LeaveGroup** (API key 13) — graceful consumer shutdown
---
### **Known Limitations**
1. **Consumer Groups:** Not implemented. Each Kafka consumer currently acts as an independent Iggy consumer. No rebalancing or offset tracking.
2. **Transactions:** Not supported. `InitProducerId`, `AddPartitionsToTxn`, and `EndTxn` return `UNSUPPORTED_VERSION`.
3. **Compression:** RecordBatch compression is preserved (opaque bytes) but not decompressed. Iggy stores the compressed payload as-is.
4. **ACLs/Auth:** The bridge authenticates to Iggy as root. Kafka client auth (SASL, mTLS) is not mapped to Iggy users.
5. **Partition Assignment:** Kafka clients cannot control which partition receives a message (balanced partitioning only).
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-16911781
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Below is the proposed architecture for the version 0.01 of kafka_2_iggy bridge. My thought process is provide a bridge for transition to iggy with the current iggy capabilities. As Iggy's new server and SDK is available, the known gaps by design will be taken care.
### **Architecture Overview**
**Approach:** **Standalone TCP Proxy** (referenced as Option 1 in discussion #3081, comment #3044)
The bridge runs as a separate process that:
1. **Listens** on port 9093 (Kafka wire protocol)
2. **Decodes** Kafka binary requests (ApiVersions, Produce, Fetch, etc.)
3. **Translates** to Iggy SDK calls via `IggyBridge`
4. **Encodes** Kafka-compliant responses and sends them back to the client
### **Key Technical Achievements**
#### **1. Complete Protocol Foundation**
**Files:** `src/protocol/codec.rs`, `src/protocol/header.rs`
Implement a zero-copy encoder/decoder for all Kafka primitives:
---
#### **2. Request/Response Header Auto-detection**
**File:** `src/protocol/header.rs`
Kafka has **two incompatible header formats**:
- **v1 (non-flexible):** `api_key | api_version | correlation_id | client_id (NULLABLE_STRING)`
- **v2 (flexible):** `api_key | api_version | correlation_id | client_id (COMPACT_NULLABLE_STRING) | tagged_fields`
The bridge **auto-detects** which format to use based on a lookup table covering all 65 API keys:
**Impact:** Clients using old versions (e.g., Kafka 2.x) and new versions (Kafka 3.x+) work seamlessly against the same bridge.
---
#### **3. Kafka-to-Iggy Mapping Layer**
**File:** `src/iggy_bridge.rs`
The `IggyBridge` struct wraps the Iggy SDK and provides **idempotent** operations:
| Kafka Concept | Iggy Mapping | Method |
|---------------|--------------|--------|
| **Topic** | Stream + Topic (same name) | `ensure_stream_and_topic()` |
| **Partition** | Balanced partitioning (Iggy decides) | `produce()` |
| **Offset** | Message offset (0-based, same as Kafka) | `fetch()` |
| **High watermark** | Current end offset | `high_watermark()` |
**Key Design Decision:** Kafka partition IDs are **ignored** during produce. The bridge uses `Partitioning::balanced()` so Iggy handles partition assignment. This avoids 0-vs-1-based index confusion and leverages Iggy's native load balancing. This implementation will be revisited after the iggy partition mapping is implemented. For now it will be same partition id.
---
#### **4. Implemented API Keys (Phase 1)**
| API Key | Name | Versions | Status | Iggy Integration |
|---------|------|----------|--------|------------------|
| **18** | ApiVersions | v0–v3 | Complete | N/A (handshake only) |
| **3** | Metadata | v0–v9 | Complete | Returns stub (single broker at 127.0.0.1:9093) |
| **0** | Produce | v3–v9 | Complete | `send_messages()` with balanced partitioning |
| **1** | Fetch | v4–v12 | Complete | `poll_messages()` with offset-based polling |
| **2** | ListOffsets | v1–v6 | Complete | `high_watermark()` for latest offset |
| **19** | CreateTopics | v2–v5 | Complete | `create_stream()` + `create_topic()` |
**Code Locations:**
- Decoders: `src/protocol/requests.rs`
- Encoders: `src/protocol/responses.rs`
- Iggy integration: `src/protocol/api.rs` (`handle_request_with_iggy()`)
---
### **5. Comprehensive Testing Infrastructure**
#### **Unit Tests**
- **Codec tests** (`tests/codec_tests.rs`): Varint encoding, nullable strings, compact arrays
- **Header tests** (`tests/header_tests.rs`): Header v1/v2 round-trip, correlation ID preservation
- **API handler tests** (`tests/api_handler_tests.rs`): ApiVersions flexible/legacy encoding
#### **Integration Tests**
- **Server lifecycle** (`tests/server_integration_tests.rs`): Connection handling, frame size limits, graceful shutdown
- **Golden wire fixtures** (`tests/golden_wire_fixtures_tests.rs`): Byte-exact validation against known-good Kafka messages
#### **End-to-End Testing Tool**
**`tools/kafka-tool`**: A CLI that generates **280+ valid Kafka wire messages** (every API key × every version) using the `kafka-protocol` crate.
**Purpose:** Ensures the bridge doesn't crash or produce malformed responses when faced with any valid Kafka request.
---
### **Technical Challenges Solved**
#### **Challenge 1: Flexible vs. Legacy Encoding**
Kafka introduced "flexible" encoding (compact arrays, tagged fields) gradually across versions. For example:
- **Produce v9+** is flexible
- **Produce v3–v8** is legacy
- **ApiVersions v3+** is flexible
- **Metadata v9+** is flexible (but v0–v8 is legacy)
**Solution:** Version-aware encoder/decoder that checks `api_version >= flexible_threshold` and calls the appropriate codec methods (`write_compact_nullable_string()` vs. `write_nullable_string()`).
---
#### **Challenge 2: RecordBatch Format**
Kafka Produce requests contain a **RecordBatch** (Kafka's internal message format) with:
- Compression (gzip, snappy, lz4, zstd)
- Timestamps (millisecond precision)
- Headers (key-value metadata)
- Checksums (CRC32C)
**Current Solution:** The bridge treats RecordBatch as **opaque bytes** and stores them directly in Iggy. This preserves the original format but means:
- Iggy can't filter on Kafka message headers
- Iggy can't decompress Kafka messages for storage optimization
- Kafka consumers can decode the RecordBatch natively
**Future Enhancement:** Parse RecordBatch and extract individual messages, storing them as Iggy messages with metadata preserved in headers.
---
#### **Challenge 3: Partition Mapping**
**Problem:** Kafka partitions are 0-based (0, 1, 2, ...), while Iggy partitions are 1-based (1, 2, 3, ...).
**Solution:** The bridge **ignores** Kafka partition IDs during produce and uses `Partitioning::balanced()`. For fetch, it polls across all partitions and returns messages in offset order. This avoids off-by-one bugs and leverages Iggy's native balancing.
**Trade-off:** Kafka clients lose control over which partition receives a message. For sticky-key partitioning (e.g., all messages for `user_id=123` go to partition 0), a future enhancement would be:
TODO:
Design for this will be revisited after the server-ng and in works features are available.
---
### **Comparison to "Built-in Listener" Approach**
Discussion #3081 mentions integrating Kafka protocol directly into Iggy server. Here's why the **standalone proxy is superior**:
| Concern | Standalone Proxy | Built-in Listener |
|---------|------------------|-------------------|
| **Iggy codebase impact** | Zero (pure client) | High (new TCP listener + dispatcher) |
| **Testing complexity** | Isolated unit tests | Requires Iggy server in test environment |
| **Deployment flexibility** | Optional (run only if Kafka compat needed) | Always-on (port 9093 always open) |
| **Version evolution** | Independent releases | Tied to Iggy release cycle |
| **Security boundary** | Proxy can enforce Kafka ACLs separately | ACLs mixed with Iggy auth |
| **Latency** | +1-2ms | ~0ms |
**Verdict:** Standalone proxy is the **pragmatic choice** for MVP. Built-in listener could be considered for v2 if latency becomes a bottleneck (unlikely for most workloads).
---
### **Next Steps (Phase 2 API Keys)**
To achieve **full producer/consumer compatibility**, the following API keys are required:
#### **Critical (Producer)**
- **Produce**
- **ApiVersions**
- **Metadata**
- **InitProducerId** (API key 22) — needed for idempotent/transactional producers
- **AddPartitionsToTxn** (API key 24) — transactional writes
- **EndTxn** (API key 26) — commit/abort transactions
#### **Critical (Consumer)**
- **Fetch**
- **ListOffsets**
- **FindCoordinator** (API key 10) — locate consumer group coordinator
- **JoinGroup** (API key 11) — join consumer group
- **SyncGroup** (API key 14) — receive partition assignment
- **Heartbeat** (API key 12) — keep group membership alive
- **OffsetCommit** (API key 8) — commit consumer offsets
- **OffsetFetch** (API key 9) — fetch last committed offsets
- **LeaveGroup** (API key 13) — graceful consumer shutdown
---
### **Known Limitations**
1. **Consumer Groups:** Not implemented. Each Kafka consumer currently acts as an independent Iggy consumer. No rebalancing or offset tracking.
2. **Transactions:** Not supported. `InitProducerId`, `AddPartitionsToTxn`, and `EndTxn` return `UNSUPPORTED_VERSION`.
3. **Compression:** RecordBatch compression is preserved (opaque bytes) but not decompressed. Iggy stores the compressed payload as-is.
4. **ACLs/Auth:** The bridge authenticates to Iggy as root. Kafka client auth (SASL, mTLS) is not mapped to Iggy users.
5. **Partition Assignment:** Kafka clients cannot control which partition receives a message (balanced partitioning only).
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-16911781
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Thanks @spetz . I was of the view , having a separate folder at the root level , parallel to core and name as bridges/gateways . With in the gateways, add kafka folder. Later redpandas,MQ etc. Each of the sub folder is a standalone application. Repository refactoring can be done on need basis.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17153999
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla edited a comment on the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Below is the proposed architecture for the version 0.01 of kafka_2_iggy bridge. My thought process is to provide a bridge for transition to iggy with the current iggy capabilities. As Iggy's new server and SDK is available, the known gaps by design will be taken care.
### **Architecture Overview**
**Approach:** **Standalone TCP Proxy** (referenced as Option 1 in discussion #3081, comment #3044)
The bridge runs as a separate process that:
1. **Listens** on port 9093 (Kafka wire protocol)
2. **Decodes** Kafka binary requests (ApiVersions, Produce, Fetch, etc.)
3. **Translates** to Iggy SDK calls via `IggyBridge`
4. **Encodes** Kafka-compliant responses and sends them back to the client
### **Key Technical Achievements**
#### **1. Complete Protocol Foundation**
**Files:** `src/protocol/codec.rs`, `src/protocol/header.rs`
Implement a zero-copy encoder/decoder for all Kafka primitives:
---
#### **2. Request/Response Header Auto-detection**
**File:** `src/protocol/header.rs`
Kafka has **two incompatible header formats**:
- **v1 (non-flexible):** `api_key | api_version | correlation_id | client_id (NULLABLE_STRING)`
- **v2 (flexible):** `api_key | api_version | correlation_id | client_id (COMPACT_NULLABLE_STRING) | tagged_fields`
The bridge **auto-detects** which format to use based on a lookup table covering all 65 API keys:
**Impact:** Clients using old versions (e.g., Kafka 2.x) and new versions (Kafka 3.x+) work seamlessly against the same bridge.
---
#### **3. Kafka-to-Iggy Mapping Layer**
**File:** `src/iggy_bridge.rs`
The `IggyBridge` struct wraps the Iggy SDK and provides **idempotent** operations:
| Kafka Concept | Iggy Mapping | Method |
|---------------|--------------|--------|
| **Topic** | Stream + Topic (same name) | `ensure_stream_and_topic()` |
| **Partition** | Balanced partitioning (Iggy decides) | `produce()` |
| **Offset** | Message offset (0-based, same as Kafka) | `fetch()` |
| **High watermark** | Current end offset | `high_watermark()` |
**Key Design Decision:** Kafka partition IDs are **ignored** during produce. The bridge uses `Partitioning::balanced()` so Iggy handles partition assignment. This avoids 0-vs-1-based index confusion and leverages Iggy's native load balancing. This implementation will be revisited after the iggy partition mapping is implemented. For now it will be same partition id.
---
#### **4. Implemented API Keys (Phase 1)**
| API Key | Name | Versions | Iggy Integration |
|---------|------|----------|--------|------------------|
| **18** | ApiVersions | v0–v3 | N/A (handshake only) |
| **3** | Metadata | v0–v9 |Returns stub (single broker at 127.0.0.1:9093) |
| **0** | Produce | v3–v9 | `send_messages()` with balanced partitioning |
| **1** | Fetch | v4–v12 | `poll_messages()` with offset-based polling |
| **2** | ListOffsets | v1–v6 | `high_watermark()` for latest offset |
| **19** | CreateTopics | v2–v5 | `create_stream()` + `create_topic()` |
**Code Locations:**
- Decoders: `src/protocol/requests.rs`
- Encoders: `src/protocol/responses.rs`
- Iggy integration: `src/protocol/api.rs` (`handle_request_with_iggy()`)
---
### **5. Comprehensive Testing Infrastructure**
#### **Unit Tests**
- **Codec tests** (`tests/codec_tests.rs`): Varint encoding, nullable strings, compact arrays
- **Header tests** (`tests/header_tests.rs`): Header v1/v2 round-trip, correlation ID preservation
- **API handler tests** (`tests/api_handler_tests.rs`): ApiVersions flexible/legacy encoding
#### **Integration Tests**
- **Server lifecycle** (`tests/server_integration_tests.rs`): Connection handling, frame size limits, graceful shutdown
- **Golden wire fixtures** (`tests/golden_wire_fixtures_tests.rs`): Byte-exact validation against known-good Kafka messages
#### **End-to-End Testing Tool**
**`tools/kafka-tool`**: A CLI that generates **280+ valid Kafka wire messages** (every API key × every version) using the `kafka-protocol` crate.
**Purpose:** Ensures the bridge doesn't crash or produce malformed responses when faced with any valid Kafka request.
---
### **Technical Challenges To Solve**
#### **Challenge 1: Flexible vs. Legacy Encoding**
Kafka introduced "flexible" encoding (compact arrays, tagged fields) gradually across versions. For example:
- **Produce v9+** is flexible
- **Produce v3–v8** is legacy
- **ApiVersions v3+** is flexible
- **Metadata v9+** is flexible (but v0–v8 is legacy)
**Solution:** Version-aware encoder/decoder that checks `api_version >= flexible_threshold` and calls the appropriate codec methods (`write_compact_nullable_string()` vs. `write_nullable_string()`).
---
#### **Challenge 2: RecordBatch Format**
Kafka Produce requests contain a **RecordBatch** (Kafka's internal message format) with:
- Compression (gzip, snappy, lz4, zstd)
- Timestamps (millisecond precision)
- Headers (key-value metadata)
- Checksums (CRC32C)
**Proposed Solution:** The bridge treats RecordBatch as **opaque bytes** and stores them directly in Iggy. This preserves the original format but means:
- Iggy can't filter on Kafka message headers
- Iggy can't decompress Kafka messages for storage optimization
- Kafka consumers can decode the RecordBatch natively
**Future Enhancement:** Parse RecordBatch and extract individual messages, storing them as Iggy messages with metadata preserved in headers.
---
#### **Challenge 3: Partition Mapping**
**Problem:** Kafka partitions are 0-based (0, 1, 2, ...), while Iggy partitions are 1-based (1, 2, 3, ...).
**Solution:** The bridge **ignores** Kafka partition IDs during produce and uses `Partitioning::balanced()`. For fetch, it polls across all partitions and returns messages in offset order. This avoids off-by-one bugs and leverages Iggy's native balancing.
**Trade-off:** Kafka clients lose control over which partition receives a message. For sticky-key partitioning (e.g., all messages for `user_id=123` go to partition 0), a future enhancement would be:
TODO:
Design for this will be revisited after the server-ng and in works features are available.
---
### **Comparison to "Built-in Listener" Approach**
Discussion #3081 mentions integrating Kafka protocol directly into Iggy server. Here's why the **standalone proxy is superior**:
| Concern | Standalone Proxy | Built-in Listener |
|---------|------------------|-------------------|
| **Iggy codebase impact** | Zero (pure client) | High (new TCP listener + dispatcher) |
| **Testing complexity** | Isolated unit tests | Requires Iggy server in test environment |
| **Deployment flexibility** | Optional (run only if Kafka compat needed) | Always-on (port 9093 always open) |
| **Version evolution** | Independent releases | Tied to Iggy release cycle |
| **Security boundary** | Proxy can enforce Kafka ACLs separately | ACLs mixed with Iggy auth |
| **Latency** | +1-2ms | ~0ms |
**Verdict:** Standalone proxy is the **pragmatic choice** for MVP. Built-in listener could be considered for v2 if latency becomes a bottleneck (unlikely for most workloads).
---
### **Next Steps (Phase 2 API Keys)**
To achieve **full producer/consumer compatibility**, the following API keys are required:
#### **Critical (Producer)**
- **Produce**
- **ApiVersions**
- **Metadata**
- **InitProducerId** (API key 22) — needed for idempotent/transactional producers
- **AddPartitionsToTxn** (API key 24) — transactional writes
- **EndTxn** (API key 26) — commit/abort transactions
#### **Critical (Consumer)**
- **Fetch**
- **ListOffsets**
- **FindCoordinator** (API key 10) — locate consumer group coordinator
- **JoinGroup** (API key 11) — join consumer group
- **SyncGroup** (API key 14) — receive partition assignment
- **Heartbeat** (API key 12) — keep group membership alive
- **OffsetCommit** (API key 8) — commit consumer offsets
- **OffsetFetch** (API key 9) — fetch last committed offsets
- **LeaveGroup** (API key 13) — graceful consumer shutdown
---
### **Known Limitations**
1. **Consumer Groups:** Not implemented. Each Kafka consumer currently acts as an independent Iggy consumer. No rebalancing or offset tracking.
2. **Transactions:** Not supported. `InitProducerId`, `AddPartitionsToTxn`, and `EndTxn` return `UNSUPPORTED_VERSION`.
3. **Compression:** RecordBatch compression is preserved (opaque bytes) but not decompressed. Iggy stores the compressed payload as-is.
4. **ACLs/Auth:** The bridge authenticates to Iggy as root. Kafka client auth (SASL, mTLS) is not mapped to Iggy users.
5. **Partition Assignment:** Kafka clients cannot control which partition receives a message (balanced partitioning only).
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-16911781
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz edited a comment on the discussion: feature - Technical implementation discussion of kafka to iggy bridge
I still need to do more reading, just wanted to clarify that Iggy partitions are also 0 based :)
Also, what does it mean that the consumer groups aren't implemented? It's been part of Iggy since its inception (almost).
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-16966781
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
I still need to do more reading, just wanted to clarify that Iggy partitions are also 0 based :)
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-16966781
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla closed a discussion: feature - Technical implementation discussion of kafka to iggy bridge
The discussion is branching out the discussion from https://github.com/apache/iggy/discussions/3081 around the architectural and design of the kafka_iggy_bridge . The design details will be added here for community discussion and sharing the implementation idea. All the implementation jira tickets will be added here.
GitHub link: https://github.com/apache/iggy/discussions/3252
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
The issue with SDKs is that it's also doubled all the work and maintenance - and I think that for most engineers out there using Kafka, they'd rather use the existing codebase, point it to "something" to either migrate/proxy/mirror traffic to Iggy, and then swap SDKs. Proxy sounds like the simplest way to begin with, as it's an isolated process to run and a single component to work on. SDK's approach would probably be a bit more efficient but requires modifying an existing codebase.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17149530
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla closed the discussion with a comment: feature - Technical implementation discussion of kafka to iggy bridge
As the implementation plan has started (refer https://github.com/apache/iggy/discussions/3253) I am closing this discussion.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17181185
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
Thanks @spetz . I considered option 2, but given number of SDKs involved, I am leaning towards option 1. Can I go ahead and start implementing option 1?
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17125639
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
I was thinking a bit more about this potential compatibility, and IMHO there are 2 ways, both of which have their pros & cons:
1. A dedicated proxy to which any Kafka-compatible client could connect that would internally translate the request into an Iggy payload and then return the response based on the Kafka API spec. This would allow them to "seamlessly" integrate existing solutions without changing their codebase. However, some of the features, such as transactions, won't work. The additional complexity may also arise in the future from the Iggy SDK changes related to upcoming VSR support (but this would need to be managed by proxy layer anyway).
2. Either Kafka to Iggy or Iggy to Kafka shim at the SDK layer - it wouldn't require any external proxy to run, but instead, the existing codebase would have to include a new dependency where, depending on the direction (app using Kafka wanting to talk to Iggy, or app using Iggy wanting to talk to Kafka), a specific feature flag would need to be enabled at the given SDK. In theory, this should boil down "only" to the translation layer within the SDK (hence no performance implications). Yet, given how many SDKs there are already for multiple programming languages, it could be quite some work when it comes to implementation & future maintenance.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17059877
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: feature - Technical implementation discussion of kafka to iggy bridge
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
GitHub user ryerraguntla added a comment to the discussion: feature - Technical implementation discussion of kafka to iggy bridge
I concur with @kparisa from the point of hosting services. Being an open source migration, offer a bridge to taste the speed of Iggy and save hosting costs with SDKs with commercial support. Professional services can be offered need basis (on a lighter note creating job opportunities in the current job market). Near term it will be a quick win with a bridge then migration with SDKs. if @spetz approves the proposed technical approach then I can work on the implementation phases with other community members.
GitHub link: https://github.com/apache/iggy/discussions/3252#discussioncomment-17152946
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org