Re: [DISCUSS] proposal: move connectors into a dedicated `iggy-connectors` repository

["hubcio (via GitHub)" <gi...@apache.org>]

GitHub user hubcio edited a discussion: [DISCUSS] proposal: move connectors into a dedicated `iggy-connectors` repository

I want to move the whole connectors subsystem (`core/connectors/`: runtime, sdk,
all sinks and sources + tests) out into its own `apache/iggy-connectors` repo. nothing
decided yet, just want to hear what people think.

the reason is CI time. our test scoping walks the workspace dependency graph, and any
change to iggy-server ends up pulling the whole rust workspace in, so every
connector gets built and tested too. the plugins are `cdylib` crates that ci
always builds, so even a one-line server change rebuilds every connector `.so`
and spins up the docker/`testcontainers` backends. we have ~16 connector crates
now (10 sinks, 4 sources), some pulling huge trees like arrow/parquet, deltalake
and cloud sdks, mongo, elasticsearch. so people working on the server keep paying
for connectors they never touched, and it gets worse every time someone adds
another heavy one.

splitting is safe because the runtime only talks to the server as a normal
client, over `iggy` and `iggy_binary_protocol`, never server internals. testing
connectors against a released server is the same thing real users do. the only
internal contract is the ffi plugin abi in `iggy_connector_sdk`, and that stays
in the new repo next to the plugins that depend on it.

what we'd get: server CI stops building and container-testing connectors, the
heavy deps leave the main build, and connectors can release on their own clock.
what we'd have to deal with: server changes that break a connector get caught
later (on a release or nightly, not in the pr); `iggy_connector_sdk` becomes a
real published crate with an abi/semver policy to keep; the crates connectors
depend on (`iggy`, `iggy_common`, `iggy_binary_protocol`, `configs`) need to be
published and pinned; and since we're incubating, a new repo means going through
asf infra.

rough order if we do it: land the open connector prs, file an apache infra
ticket to create the `apache/iggy-connectors` repo, split with history into it,
publish and pin the dep crates, set up ci that runs against the latest released
server, then drop `core/connectors/` from the main workspace.

a few of us already like the idea, but i'd rather argue it out here first. if
you maintain or run a connector, or you think this is a bad move, say so.


GitHub link: https://github.com/apache/iggy/discussions/3580

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org