You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "mmmmxa (via GitHub)" <gi...@apache.org> on 2026/10/13 15:29:15 UTC
[PR] feat(connectors): ship connector plugins in the iggy-connect image [iggy]
mmmmxa opened a new pull request, #4533:
URL: https://github.com/apache/iggy/pull/4533
## Which issue does this PR address?
Closes #3614
Continues #3658 by @kriti-sc, which went stale. Co-authored by @kriti-sc.
## Rationale
Trying out a connector meant compiling the plugin from source and wiring its path by hand, because the `apache/iggy-connect` image shipped only the runtime. Shipping the plugins in the image, together with a compose example that works out of the box, removes that friction.
## What changed?
The `iggy-connect` image contained only the runtime binary, so every plugin had to be built and mounted separately. It is now published in two flavors: a fat default image with the runtime and every connector plugin in `/usr/local/lib`, loadable by file name, and a `-slim` image with the runtime only. The plugin set is derived from `cargo metadata` in `scripts/ci/connector-plugins.sh`, and the image build, edge tarball, release notes and license checks all use it.
The `sink-data-producer` example is now a Delta Lake sink demo. `docker compose up -d --build` starts the server, the connectors runtime and floci (an S3 emulator) and creates the stream, topic, bucket and Delta table. `cargo run --example sink-data-producer` produces the data, and two commands show it in the table. It replaces the Quickwit quick start in the connectors README and runs correctly end to end, which removes the main friction in developing connectors.
### Changes from the review of #3658
Fixed:
- The slim image no longer compiles plugins, a failed plugin copy now fails the build, and the platform mapping is defined once.
- `reqwest`, `tracing-subscriber` and `serde_json` features are pinned at the workspace root, so `-p` builds keep the features of the old `--bin` build.
- The fat `LICENSE-binary` and the CI license check scan the runtime and each plugin separately against the real `Cargo.lock` and merge the results. Before, the runtime's own dependencies were missing.
- The build cache is scoped per flavor. One manifest job per component pushes every flavor and creates the git tag last, so a failed slim push cannot leave a tag behind. Flavor config is validated, the Dockerfile stage field is called `stage`, `.dockerignore` is part of the `use_latest_ci` overlay, and job names include the flavor.
- Images carry an `org.apache.iggy.flavor` label.
- The release notes list the plugins found in the shipped tarball, the unused `connector_plugins` input is gone, and the edge image variants track the plugin crates and `connector-plugins.sh`.
- A `docker-connectors-fat` pre-merge task builds the fat image when the Dockerfile, plugin manifests or their build scripts change.
- `connector-plugins.sh` filters on the `iggy_connector_` prefix and warns about skipped cdylibs, and the README plugin lists include the connectors that were missing.
Not taken as suggested:
- `--package-flags` stays in `connector-plugins.sh` because the Dockerfile planner still uses it.
- The label key is `org.apache.iggy.flavor`, not `org.opencontainers.image.variant`. The `org.opencontainers.image` namespace is reserved by the OCI spec, and `variant` already means the CPU variant.
- Excluding docs-only edits from the edge gate no longer applies, because master replaced the path gate with `edge-image-variants.json`.
## Local Execution
- Passed. Built the fat image locally and ran the compose example end to end.
- Pre-commit hooks ran (`prek`).
## AI Usage
1. Claude (Claude Code).
2. I worked through the review comments of #3658 with it. It implemented the approved fixes, the compose example, the example configs and helper scripts, and the README changes. I chose the designs and reviewed every diff.
3. Ran the example end to end locally. Format, lint and config checks ran through `prek`.
4. Yes.
Co-authored-by: kriti-sc <38...@users.noreply.github.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): ship connector plugins in the iggy-connect image [iggy]
Posted by "mmmmxa (via GitHub)" <gi...@apache.org>.
mmmmxa commented on PR #4533:
URL: https://github.com/apache/iggy/pull/4533#issuecomment-6149347119
/skill team-review-slim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): ship connector plugins in the iggy-connect image [iggy]
Posted by "mmodzelewski (via GitHub)" <gi...@apache.org>.
mmodzelewski commented on code in PR #4533:
URL: https://github.com/apache/iggy/pull/4533#discussion_r4389765142
##########
core/connectors/runtime/Dockerfile:
##########
@@ -75,41 +116,117 @@ COPY . .
RUN --mount=type=cache,target=/usr/local/cargo/registry,id=cargo-registry-${TARGETPLATFORM}-${LIBC} \
--mount=type=cache,target=/usr/local/cargo/git,id=cargo-git-${TARGETPLATFORM}-${LIBC} \
--mount=type=cache,target=/app/target,id=cargo-target-${TARGETPLATFORM}-${LIBC} \
- case "$TARGETPLATFORM:$LIBC" in \
- "linux/amd64:musl") RUST_TARGET="x86_64-unknown-linux-musl" ;; \
- "linux/arm64:musl") RUST_TARGET="aarch64-unknown-linux-musl" ;; \
- "linux/amd64:glibc") RUST_TARGET="x86_64-unknown-linux-gnu" ;; \
- "linux/arm64:glibc") RUST_TARGET="aarch64-unknown-linux-gnu" ;; \
- *) echo "Unsupported platform/libc combination: $TARGETPLATFORM/$LIBC" && exit 1 ;; \
- esac && \
+ RUST_TARGET="$(cat /rust-target)" && \
+ if [ "$PROFILE" = "debug" ]; then PROFILE_DIR=debug; PROFILE_FLAG=""; else PROFILE_DIR=release; PROFILE_FLAG="--release"; fi && \
+ cargo zigbuild --locked --target ${RUST_TARGET} --bin iggy-connectors $PROFILE_FLAG && \
+ cp /app/target/${RUST_TARGET}/${PROFILE_DIR}/iggy-connectors /app/iggy-connectors
+
+#
+# Generate the third-party license manifest. Required by ASF release policy:
+# convenience binaries that statically link third-party code MUST include the
+# full license text of every bundled crate inside the artifact. This one covers
+# the runtime only, which is all the slim image ships.
+#
+RUN --mount=type=cache,target=/usr/local/cargo/registry,id=cargo-registry-${TARGETPLATFORM}-${LIBC} \
+ --mount=type=cache,target=/usr/local/cargo/git,id=cargo-git-${TARGETPLATFORM}-${LIBC} \
+ ./scripts/ci/third-party-licenses.sh --generate --manifest core/connectors/runtime/Cargo.toml --output /app/LICENSE-binary
+
+#
+# Fat builder - runtime binary plus every connector plugin cdylib, and a
+# license manifest covering all of them.
+#
+FROM --platform=$BUILDPLATFORM toolchain AS fat-builder
+ARG PROFILE=release
+ARG TARGETPLATFORM
+ARG LIBC=glibc
+
+# Only this branch reads the derived plugin set, so adding a connector leaves
+# the slim branch's cooked dependency layer untouched. Two lists, because cargo
+# selects a plugin by package name while the artifact is lib<target name>.so.
+COPY --from=planner /app/connector-plugins.txt connector-plugins.txt
+COPY --from=planner /app/connector-packages.txt connector-packages.txt
+COPY --from=planner /app/connector-manifests.txt connector-manifests.txt
+
+#
+# Cook dependencies
+#
+RUN --mount=type=cache,target=/usr/local/cargo/registry,id=cargo-registry-${TARGETPLATFORM}-${LIBC} \
+ --mount=type=cache,target=/usr/local/cargo/git,id=cargo-git-${TARGETPLATFORM}-${LIBC} \
+ --mount=type=cache,target=/app/target,id=cargo-target-fat-${TARGETPLATFORM}-${LIBC} \
+ RUST_TARGET="$(cat /rust-target)" && \
+ PLUGIN_FLAGS="$(cat connector-packages.txt)" && \
if [ "$PROFILE" = "debug" ]; then \
- cargo zigbuild --locked --target ${RUST_TARGET} --bin iggy-connectors && \
- cp /app/target/${RUST_TARGET}/debug/iggy-connectors /app/iggy-connectors; \
+ cargo chef cook --recipe-path recipe.json --target ${RUST_TARGET} --zigbuild \
+ --bin iggy-connectors && \
+ cargo chef cook --recipe-path recipe.json --target ${RUST_TARGET} --zigbuild \
+ $PLUGIN_FLAGS; \
else \
- cargo zigbuild --locked --target ${RUST_TARGET} --bin iggy-connectors --release && \
- cp /app/target/${RUST_TARGET}/release/iggy-connectors /app/iggy-connectors; \
+ cargo chef cook --recipe-path recipe.json --target ${RUST_TARGET} --zigbuild --release \
+ --bin iggy-connectors && \
+ cargo chef cook --recipe-path recipe.json --target ${RUST_TARGET} --zigbuild --release \
+ $PLUGIN_FLAGS; \
fi
+COPY . .
+
#
-# Generate third-party license manifest. Required by ASF release policy:
-# convenience binaries that statically link third-party code MUST include
-# the full license text of every bundled crate inside the artifact.
+# Build
#
RUN --mount=type=cache,target=/usr/local/cargo/registry,id=cargo-registry-${TARGETPLATFORM}-${LIBC} \
--mount=type=cache,target=/usr/local/cargo/git,id=cargo-git-${TARGETPLATFORM}-${LIBC} \
- TARGET="$(uname -m)-unknown-linux-musl" && \
- curl -sSfL "https://github.com/EmbarkStudios/cargo-about/releases/download/0.9.0/cargo-about-0.9.0-${TARGET}.tar.gz" \
- | tar -xz -C /usr/local/bin --strip-components=1 "cargo-about-0.9.0-${TARGET}/cargo-about" && \
- ./scripts/ci/third-party-licenses.sh --generate --manifest core/connectors/runtime/Cargo.toml --output /app/LICENSE-binary
+ --mount=type=cache,target=/app/target,id=cargo-target-fat-${TARGETPLATFORM}-${LIBC} \
+ RUST_TARGET="$(cat /rust-target)" && \
+ PLUGIN_FLAGS="$(cat connector-packages.txt)" && \
+ if [ "$PROFILE" = "debug" ]; then PROFILE_DIR=debug; PROFILE_FLAG=""; else PROFILE_DIR=release; PROFILE_FLAG="--release"; fi && \
+ cargo zigbuild --locked --target ${RUST_TARGET} --bin iggy-connectors $PROFILE_FLAG && \
+ cargo zigbuild --locked --target ${RUST_TARGET} $PLUGIN_FLAGS $PROFILE_FLAG && \
+ cp /app/target/${RUST_TARGET}/${PROFILE_DIR}/iggy-connectors /app/iggy-connectors && \
+ mkdir -p /app/plugins && \
+ while IFS= read -r name; do \
+ [ -z "$name" ] && continue; \
+ so_file="/app/target/${RUST_TARGET}/${PROFILE_DIR}/lib${name}.so"; \
+ if [ ! -f "$so_file" ]; then echo "expected plugin artifact missing: $so_file" >&2 && exit 1; fi; \
+ cp "$so_file" /app/plugins/ || exit 1; \
+ done < connector-plugins.txt
#
-# Final runtime - Debian trixie Slim
+# Generate the third-party license manifest covering the runtime plus every
+# bundled plugin closure. Same ASF policy as the slim manifest above; each
+# image ships the manifest that matches what it actually bundles, so the slim
+# image never over-declares plugin dependencies it does not carry.
#
-FROM debian:trixie-slim AS runtime
+RUN --mount=type=cache,target=/usr/local/cargo/registry,id=cargo-registry-${TARGETPLATFORM}-${LIBC} \
+ --mount=type=cache,target=/usr/local/cargo/git,id=cargo-git-${TARGETPLATFORM}-${LIBC} \
+ MANIFEST_FLAGS="$(sed 's/^/--manifest /' connector-manifests.txt | tr '\n' ' ')" && \
+ ./scripts/ci/third-party-licenses.sh --generate --manifest core/connectors/runtime/Cargo.toml $MANIFEST_FLAGS --output /app/LICENSE-binary-fat
+
+#
+# Runtime base - Debian trixie Slim, shared by both published flavors.
+#
+FROM debian:trixie-slim AS runtime-base
Review Comment:
Critical: `runtime-base` installs no `ca-certificates`, so reqwest 0.13 `build()` fails with "No CA certificates were loaded from the system", even for `http://` URLs. Seven plugins, `http_sink` among them, fail `open()`, and `bigquery_sink` panics and aborts the runtime. Install `ca-certificates` with `--no-install-recommends` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): ship connector plugins in the iggy-connect image [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4533:
URL: https://github.com/apache/iggy/pull/4533#issuecomment-6152846442
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4533?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 88.72%. Comparing base ([`a89b26a`](https://app.codecov.io/gh/apache/iggy/commit/a89b26a4a441bf09cfa1930f7b6648f859cb57e9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a4c1cff`](https://app.codecov.io/gh/apache/iggy/commit/a4c1cff74e7470765d989e75a22d9aa3240312c2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4533 +/- ##
============================================
+ Coverage 88.25% 88.72% +0.46%
Complexity 1726 1726
============================================
Files 1316 1316
Lines 266111 266111
Branches 224746 224746
============================================
+ Hits 234863 236102 +1239
+ Misses 25731 24432 -1299
- Partials 5517 5577 +60
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.95% <ø> (-0.02%)` | :arrow_down: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.18% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.09% <ø> (+13.16%)` | :arrow_up: |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.43% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.84% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `95.05% <ø> (-0.09%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.39% <ø> (ø)` | |
| [C++ SDK](https://app.codecov.io/gh/apache/iggy/pull/4533/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.43% <ø> (ø)` | |
[see 88 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4533/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org