You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ryerraguntla (via GitHub)" <gi...@apache.org> on 2026/06/04 13:15:34 UTC
[I] feat(gateways): Build Gateway to Iggy for existing Kafka users -Listener and Core Produces/Fetch [iggy]
ryerraguntla opened a new issue, #3421:
URL: https://github.com/apache/iggy/issues/3421
### Description
Implement a foundation layer for the gateway for Kafka users to Iggy. The foundation consists of a TCP Listener receiving Kafka messages and acknowledges the messages for each of the scoped Kafka API key and respective version.
Kafka API key and versions are pretty comprehensive list. In this ticket, define the scope of supporting Kafka API keys and associated versions for each of the API Key.
The TCP Listener receives the message , validates the Kafka message format for each of the API Key and associated version.
### Affected area / component
Other / not sure
### Proposed solution
- Listens on predefined (port 9093 which is Kafka wire protocol)
- Decodes Kafka binary requests (ApiVersions, Produce, Fetch, etc.)
- Validates against the scoped API Keys and versions and the respective message formats
- Replies with return code. In case of expected data, for this ticket return predefined values.
To Do: Attach the scoped Kafka API keys and Versions.
### Alternatives considered
Other considered approaches -
1. A layer on Iggy server with a dedicated port
2. Changes to each of the SDK so that Kafka clients can natively implement.
For (1) - Iggy server path will be affected and opening an additional port at this point of product life cycle makes product vulnerable and is discouraged.
For (2) - As users are spread across multiple SDKs, the changes need to be made in many SDKs. Also existing Kafka clients has learning curve in implementing the Iggy Binary protocol.
The above approach of having a gateway for Kafka users into Iggy eliminates the above issues. The con on having a separate gateway increases hosting cost. Alternate solutions can be considered to handle the hosting costs at a later point of time.
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
### Good first issue
- [x] I think this could be a good first issue for a new contributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka gateway: Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal closed issue #3421: Kafka gateway: Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users
URL: https://github.com/apache/iggy/issues/3421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4743690459
>> @spetz @numinnex any opinion on this? - Could you chime in , so that I can take either one of the paths
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4709847346
couple more things, on your actual workflow questions (separate from the layout note above):
branch model: i'm fine with either an integration branch or sequential prs straight into master, let me get the rest of the core team's read before we lock one in. two practical notes either way: a long-lived integration branch on apache/iggy needs write/force-push rights for the rebases, which a fork contributor doesn't have, and our pre-merge ci only triggers on prs into `master` (`branches: [master]`), so sub-prs into the branch wouldn't be gated automatically. direct-to-master incremental avoids both, and it's safe here since the crate is a leaf behind an opt-in binary so master stays green between slices. branch name's a non-issue, we can settle it if we end up going that route. @spetz @numinnex any opinion on this?
pr size/order: no fixed loc cap from my side. just break it into independently-reviewable, ci-green slices and propose the boundaries, we'll iterate from there.
naming: heads-up that `gateway` isn't in the allowed pr-title scopes yet (`.github/workflows/pr-title.yml`), so add it there in the foundation pr and title things `feat(gateway): ...`. for the binary, `iggy-kafka-gateway` fits the `iggy-*` convention (iggy-server, iggy-connectors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4709599634
the one thing that actually matters for layout is workspace membership: it has to be listed in the root `members` and pull the sdk via `iggy = { workspace = true }`. that keeps it on a single lockfile with everything else (no second `Cargo.lock` that can quietly drift from the edge sdk it's tracking) and inside the same fmt/clippy/sort/test pipeline as the rest of the repo.
the directory itself i'm not fussed about, a top-level `gateways/kafka/` is fine. just keep it a single flat crate to start (lib + thin bin, `publish = false`), we can grow it from there.
it shouldn't live in `core/connectors/` though, even if it looks adjacent. connectors are dlopened `.so` plugins behind a stable ffi abi, whereas this is a standalone process speaking the kafka wire protocol and calling iggy over the network. different shape entirely.
one heads-up for the first pr:CIi components are matched by explicit path globs in `.github/config/components.yml` and there's no catch-all, so you'll need to add a `rust-gateway` component (`paths: ["gateways/**"]`, `depends_on: [rust-sdk]`) and wire it into the `rust` aggregate. otherwise gateway-only prs match no component and quietly skip clippy/fmt/sort/test, which bites post-merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4762056835
@hubcio @spetz @numinnex Depending on the longevity of the branch and considering the potential drift from the master , effort needed to reconcile from branch to master, I considered raising PR against the master should be ok. I will make sure the time taken for CI is still in the current way without adding extra dependencies and additional complexity. With this I raised the initial PR #3519 . Though I started on the next iteration of code, I will wait till this PR is approved and merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4629455719
I dont think that this is good first issue, other than that green light from my side. At the beginning we want it to be a separate binary, not part of iggy server. is that fine @ryerraguntla ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4643386138
@hubcio Thats correct. It is a standalone process , interfacing Kafka client and iggy server. This initial ticket is a bare bones TCP listener, gets kafka message, if it is supported in this scope returns an acknowledgement or response. Later tickets will implement each of the functionality end to end in phases. The proposed code structure looks something like this -
<img width="490" height="628" alt="Image" src="https://github.com/user-attachments/assets/fcc067e4-bb46-414c-8772-904dab44e938" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4757584803
Adjusting the naming convention should be iggy_gateways_kafka (plural gateway) followed by component, which follows the iggy_connectors_* and iggy_sdk_*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(gateways): Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on issue #3421:
URL: https://github.com/apache/iggy/issues/3421#issuecomment-4675165612
@spetz @hubcio @mmodzelewski @krishvishal @kparisa - I need your opinion and guidance on the below proposal. If it is acceptable, could one of you do the needful.
## Proposed integration branch for multi-PR Kafka gateway work
Following up on #3421 — the Kafka gateway work is intentionally split across **multiple successive PRs** (foundation → Iggy bridge → additional APIs/behavior) like mentioned at https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030 . Each slice should stay reviewable on its own, but the community should be able to see **cumulative progress** in one place before anything lands on `master`.
### Request
Could a maintainer please:
1. **Create an integration branch** on `apache/iggy`, e.g. `iggy_gateway_for_kafka`, from current `master`.
2. **Confirm** that sub-PRs for this effort should target that branch (not `master`) until the feature set is complete.
3. **Confirm branch naming** — happy to use `iggy_gateway_for_kafka` or whatever fits project convention.
### Proposed workflow
```
master (stable)
│
└── iggy_gateway_for_kafka ← integration branch (public, long-lived)
│
├── PR1: TCP listener, wire codec, version firewall,
│ stub responses, tests, kafka-tool
│
├── PR2: … (next slice — TBD with maintainers)
│
├── PR3: …
│
└── Final PR: iggy_gateway_for_kafka → master
(when complete; then delete branch)
```
- Each sub-PR: **one purpose**, linked to **#3421**, with local checks per `CONTRIBUTING.md`.
- After each merge, `iggy_gateway_for_kafka` advances so anyone can browse the full in-progress tree.
- **Final PR** merges the integration branch into `master` once functionality and tests are in place.
### PR1 scope (ready / nearly ready)
First sub-PR (~foundation from #3421):
- `gateways/kafka/` — TCP listener, request decode, version firewall, stub responses (Produce, Fetch, ListOffsets, Metadata, ApiVersions, CreateTopics)
- Version-aware error responses (not bare 2-byte stubs for scoped APIs)
- `KAFKA_ADVERTISED_HOST` / broker advertisement validation
- Regression tests (103 tests) + `kafka-message-gen` tool for wire fixtures
- Docs: `SCOPE.md`, `TEST_SUITE.md`, `MANUAL_TESTING.md`
Does **not** include Iggy backend bridge yet — that would be a follow-on PR on the same integration branch.
### Questions for maintainers
1. Is an **`apache/iggy` integration branch** acceptable for this feature, or do you prefer sequential PRs directly to `master`?
2. Preferred branch name — `iggy_gateway_for_kafka` or other?
3. Should **pre-merge CI** run on PRs targeting the integration branch? Today workflows appear limited to PRs into `master`; I can run full local checks on each sub-PR regardless.
4. Any guidance on **PR size/order** beyond CONTRIBUTING (e.g. <500 LOC for new contributors, high-risk protocol areas)?
### My setup
- Working branch locally: `feat(gateways)/kafka_to_iggy_listener`
- Fork: `ryerraguntla/iggy`
- Once the integration branch exists, I'll open **PR1** as:
`ryerraguntla:<branch>` → `apache/iggy:iggy_gateway_for_kafka`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org