You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "jayakasadev (via GitHub)" <gi...@apache.org> on 2026/05/08 20:08:59 UTC
[I] Add optional S3-compatible object storage as primary persistence backend [iggy]
jayakasadev opened a new issue, #3228:
URL: https://github.com/apache/iggy/issues/3228
## Description
iggy currently persists everything to local disk: per-partition append-only \`.log\` segments, sparse \`.index\` files, an append-only state log, per-consumer offset files, system info, and tokens. This issue proposes an opt-in mode where an S3-compatible object store is the only persistence medium — useful for ephemeral / scale-to-zero compute, durable-by-default archive, and deployments where local NVMe provisioning is the bottleneck.
Local-disk mode is unchanged and remains the default; the S3 backend ships behind a default-off cargo feature so fs-only deployments aren't affected.
## Component
Iggy server
## Proposed solution
A new \`ObjectStorage\` trait abstracts persistence, with a 10-phase incremental rollout. Each phase migrates one persistence subsystem onto the trait and is independently mergeable.
The S3 client is **compio-native**: built on \`rusty-s3\` (sans-IO SigV4 + request shaping) + \`cyper\` (compio HTTP client, rustls TLS). Critically, this avoids reintroducing tokio into the data path that #2020 just removed.
### Phase plan
| Phase | Scope |
|-------|-------|
| 0 | Pre-flight feasibility spike — validate rusty-s3 + cyper + compio + rustls against real AWS S3. **Done — 5/5 scenarios passed.** |
| 1 | \`ObjectStorage\` trait + \`CompioFsStorage\` + \`InMemoryStorage\` (test) + \`S3Storage\` (feature-gated) + \`BufferedMultipartWriter\`. Seam only — no production callers yet. ← **this issue's first deliverable** |
| 2 | State log + \`FileSystemInfoStorage\` + tokens onto the trait (journal + snapshot model on object backend). |
| 3 | Segment writes via multipart upload. |
| 4 | Segment reads via ranged GET + LRU byte cache (active-segment reads keep using the in-flight buffer). |
| 5 | Bootstrap, directory ops, per-partition versioned manifests for fast boot. |
| 6 | Consumer offsets repacked into one binary object per partition. |
| 7 | Retention + segment deletion. |
| 8 | Per-partition lease object (S3 conditional PUT) for split-brain safety. |
| 9 | Hardening — retries, Prometheus metrics, IAM template, perf benchmarks. |
| 10 | Documentation, sample config, release notes. |
The S3 backend ships behind a default-off \`object-storage\` cargo feature, so fs-only deployments don't pull \`rusty-s3\` / \`cyper\` / \`url\` into their dependency graph.
### Phase 0 spike outcome
A throwaway feasibility spike (~330 LoC) ran against real AWS S3 in an ephemeral bucket (us-east-1; 1-day lifecycle backstop; bucket torn down on exit). All five scenarios passed:
| Scenario | Latency |
|---|---|
| PUT 1 KiB | 108 ms |
| Range-GET 256 B | 33 ms |
| Multipart 12 MiB upload (3 parts) | 1555 ms |
| Full GET + byte-compare 12 MiB | 919 ms |
| Conditional PUT race (\`If-None-Match: *\`) | 62 ms; loser fenced cleanly with HTTP 412 |
Three correctness findings, baked into Phase 1:
1. rustls 0.23 needs an explicit \`CryptoProvider::install_default()\` when cyper is configured \`default-features = false\`.
2. AWS ETags arrive wrapped in quotes; \`rusty-s3\`'s \`complete_multipart_upload\` re-wraps them when serializing the XML body, so the ETag must be \`.trim_matches('"')\`-stripped before being passed back. Otherwise: \`400 InvalidPart\`.
3. Multipart minimum part size is 5 MiB except the final part; iggy's typical sub-MiB flushes need a buffering layer (\`BufferedMultipartWriter\`) to coalesce them into legal parts.
### Phase 1 PRs
- #3226 — Phase 1a: \`ObjectStorage\` trait + \`CompioFsStorage\` + \`InMemoryStorage\` + \`[system.storage]\` config + bootstrap seam. No \`rusty-s3\` / \`cyper\` deps yet. No-op for fs deployments.
- #3227 — Phase 1b (stacks on #3226): \`S3Storage\` + \`BufferedMultipartWriter\` behind the \`object-storage\` cargo feature. Includes an \`IGGY_TEST_MINIO\`-gated wire test.
## Alternatives considered
1. **Build on \`opendal\` instead of \`rusty-s3\` + \`cyper\`.** Tempting because opendal supports more backends out-of-the-box (GCS native, Azure native), but opendal is tokio-native. Pulling it in would require either reintroducing tokio into iggy's data-path runtime (undoing #2020) or running opendal behind a per-call thread bridge (channel-hop overhead per S3 call, plus an extra runtime). Rejected.
2. **Use \`rust-s3\` (already a transitive dep via the iceberg sink).** Also tokio-based via reqwest. Same problem. Left in place for the existing iceberg connector; not used for the new path.
3. **Roll a thin compio-native HTTP client over \`compio::net::TcpStream\` + \`rustls\` directly.** Minimal external surface but ~300 LoC of HTTP/SigV4 plumbing to maintain in-tree. Reserved as a fallback if \`cyper\` later turns out to have sharp edges; the Phase 0 spike confirmed it doesn't, today.
## Open questions for maintainers
- **Issue cadence.** Happy to file separate issues per phase (one-issue-one-PR) if you prefer that to a single umbrella. This issue is intended to anchor design discussion for the milestone; each phase still ships in its own PR(s).
- **Default \`multipart_part_size\`.** Currently 8 MiB (configurable). AWS minimum is 5 MiB except final. Smaller → finer durability + more S3 PUTs; larger → fewer PUTs + larger memory buffers. 8 MiB is a starting guess; alternatives 5 / 16 / 32.
- **\`ack_after_upload\` default.** True (producer ack waits for part-upload success — durable before producer learns) is the safe default. False is faster but loses messages on crash before next flush; intended for testing only. Reasonable?
- **GCS-S3-compat support.** Phase 8 (fencing) uses S3 \`If-None-Match: *\` conditional PUT. AWS / MinIO / R2 / Tigris support this; GCS-S3-compat does **not**. OK to document GCS as not-supported in Phase 8 and revisit later, or should we adopt a different fencing mechanism (paid coordination service)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add optional S3-compatible object storage as primary persistence backend [iggy]
Posted by "jayakasadev (via GitHub)" <gi...@apache.org>.
jayakasadev commented on issue #3228:
URL: https://github.com/apache/iggy/issues/3228#issuecomment-4413730814
I appreciate the feedback. I'll defer for now and keep an eye on `core/server-ng`.
I can create a discussion for now and add a note that this is future work.
my discord handle is jaykay2011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add optional S3-compatible object storage as primary persistence backend [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #3228:
URL: https://github.com/apache/iggy/issues/3228#issuecomment-4412311425
hey @jayakasadev, thanks for the writeup and the spike.
process note first: a feature this size (new persistence backend, 10 phases, new deps, fencing semantics) should land as a GitHub Discussion before an issue + stacked PRs. the core team needs a chance to weigh in on the design before code shows up. please open one for this so the rest of the maintainers can review it.
important note: `core/server` is being replaced by `core/server-ng`, which lives alongside it during the cluster (based on VSR) buildout. server-ng is still early and the immediate roadmap is finishing consensus + cutting the cluster. landing a 10-phase persistence rewrite into `core/server` right now means phases 2 through 10 ship into code we are actively deprecating. net loss for everyone, including you.
so the call: defer until cluster has merged (and is fully functional) and server-ng is the live target. once that lands, the discussion above can drive whether this design is the one we adopt, or whether we want a different shape.
#3226 / #3227: we will close them. once design is approved in the discussions, we can proceed with the implementation, not the other way around.
just to add: work is not wasted, just early.
also, have you already joined our discord? if yes, then whats your handle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add optional S3-compatible object storage as primary persistence backend [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio closed issue #3228: Add optional S3-compatible object storage as primary persistence backend
URL: https://github.com/apache/iggy/issues/3228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org