Design proposal: S3 as optional primary storage backend (future work, server-ng-targeted)

["jayakasadev (via GitHub)" <gi...@apache.org>]

GitHub user jayakasadev created a discussion: Design proposal: S3 as optional primary storage backend (future work, server-ng-targeted)

## Status

This is a **design proposal for future work** targeting `core/server-ng` once cluster + consensus have landed. **Not** a request to implement against `core/server`.

Filed at the suggestion of @hubcio in #3228: "a feature this size should land as a GitHub Discussion before an issue + stacked PRs." PRs #3226 / #3227 are closed; this Discussion is the design-level reset.

## Goal

Allow iggy to run with an S3-compatible object store as the *only* persistence medium — no local data files. Useful for ephemeral / scale-to-zero compute, durable-by-default archive, and deployments where local NVMe provisioning is the bottleneck. Local-disk mode remains the default; S3 mode is opt-in via config and behind a default-off cargo feature.

## Why now (and why "future work")

`core/server` is being replaced by `core/server-ng` (VSR-based). Landing a 10-phase persistence rewrite into `core/server` ships into actively-deprecated code. So this proposal targets the *post*-server-ng codebase, after consensus + cluster have merged. Filing the design now gives the core team time to weigh in on shape so implementation isn't bottlenecked on design when server-ng is ready.

## High-level shape

A small `ObjectStorage` trait abstracts the persistence subsystem:

```rust
#[async_trait(?Send)]
pub trait ObjectStorage: Debug {
    async fn put(&self, key: &str, bytes: Bytes) -> Result<(), IggyError>;
    async fn put_if_absent(&self, key: &str, bytes: Bytes) -> Result<(), IggyError>;
    async fn put_multipart(&self, key: &str) -> Result<Box<dyn MultipartHandle>, IggyError>;
    async fn get_range(&self, key: &str, range: Range<u64>) -> Result<Bytes, IggyError>;
    async fn head(&self, key: &str) -> Result<ObjectMeta, IggyError>;
    async fn list_prefix(&self, prefix: &str) -> Result<Vec<ObjectMeta>, IggyError>;
    async fn delete(&self, key: &str) -> Result<(), IggyError>;
    async fn delete_many(&self, keys: &[&str]) -> Result<(), IggyError> { /* default per-key */ }
}
```

Three backends:
- `CompioFsStorage` — passthrough to compio fs (current behavior)
- `S3Storage` — built on `rusty-s3` (sans-IO SigV4) + `cyper` (compio HTTP, rustls TLS), behind the `object-storage` cargo feature
- `InMemoryStorage` — `cfg(test)`, HashMap-backed (replaces what `opendal::services::Memory` would have given us)

**No tokio. No opendal.** Compio-native end-to-end so it doesn't undo the work in #2020 / #2247.

## Phase 0 spike (already run, against real AWS S3)

A throwaway feasibility spike validated the rusty-s3 + cyper + compio + rustls combination against real AWS S3 in an ephemeral bucket (us-east-1, 1-day lifecycle backstop, torn down on exit). 5/5 scenarios passed:

| Scenario | Latency |
|---|---|
| PUT 1 KiB | 108 ms |
| Range-GET 256 B | 33 ms |
| Multipart 12 MiB upload (3 parts) | 1555 ms |
| Full GET + byte-compare 12 MiB | 919 ms |
| Conditional PUT race (`If-None-Match: *`) | 62 ms; loser fenced cleanly with HTTP 412 |

Three correctness findings:
1. `rustls::crypto::ring::default_provider().install_default()` is required at boot when cyper is configured `default-features = false`.
2. AWS ETags arrive wrapped in quotes; `rusty-s3`'s `complete_multipart_upload` re-wraps them, so callers must `.trim_matches('"')`-strip before passing back. Otherwise: `400 InvalidPart`.
3. Multipart minimum part size is 5 MiB except the final part — typical iggy sub-MiB flushes need a buffering layer (`BufferedMultipartWriter`) to coalesce them into legal parts.

## Open questions for server-ng integration

The trait + S3 backend + buffering + credential chain are backend-agnostic. The questions are about *where* they plug in:

1. **Persistence model.** Does server-ng keep a per-shard local log + segments + state log, or does VSR's replicated log replace the state log entirely?
2. **State log.** The proposal had a journal+snapshot model on object storage (since S3 has no append). With VSR, is the state log even a per-replica concept? Or does the consensus layer own it?
3. **Segment writes / reads.** Where does the `MessagesWriter` / `MessagesReader` shape live in server-ng? Is the multipart-during-write model still the right fit, or does VSR's commit model change the durability story?
4. **Manifest model.** The proposal had a per-partition versioned manifest for fast boot. With VSR, the consensus state may already enumerate segments — manifest redundant?
5. **Fencing.** Original Phase 8 used S3 conditional PUT for split-brain safety. VSR view-changes provide single-writer-per-view by construction. **Probably retires entirely** in a server-ng world. Does that match your view? (Side benefit: also retires the GCS-S3-compat limitation since there's no `If-None-Match` requirement anymore.)
6. **`ack_after_upload` semantics.** Producer ack waits for S3 PUT success. VSR adds a quorum-ack on top. Which is the user-visible commit point?
7. **Trait ownership.** Is `Rc<dyn ObjectStorage>` (per-shard) the right model for server-ng, or does it want `Arc<dyn ObjectStorage>` shared more broadly?

## AWS credential chain (already designed)

Subset of the standard provider chain, no AWS SDK dep, all on cyper:
1. Inline config (`[system.storage.object]` access_key_id + secret_access_key)
2. Standard env vars (`AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` + optional `AWS_SESSION_TOKEN`)
3. Container credentials (`AWS_CONTAINER_CREDENTIALS_FULL_URI` / `RELATIVE_URI` — covers ECS task roles + EKS Pod Identity)
4. IRSA via STS `AssumeRoleWithWebIdentity` (covers EKS IRSA)

Credential refresh + IMDSv2 deferred to a later "hardening" phase.

## What I'd like from this Discussion

- Direction on which of the open questions above matter most
- Any shape constraints from the in-progress server-ng / VSR / cluster work that should inform the trait surface
- Whether the design as proposed is broadly the one you'd want, or whether server-ng wants a different shape
- A signal on when this work could realistically resume (i.e., when server-ng is the live target)

Happy to revise based on any of the above.

---

**References to closed artifacts:** PRs #3226 / #3227 and issue #3228 contain the original (server-targeted) design and Phase 0 spike findings.


GitHub link: https://github.com/apache/iggy/discussions/3232

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: S3 as optional primary storage backend (future work, server-ng-targeted)

["jiengup (via GitHub)" <gi...@apache.org>]

GitHub user jiengup added a comment to the discussion: Design proposal: S3 as optional primary storage backend (future work, server-ng-targeted)

@jayakasadev 
Hi, it's a very nice proposal and a very convincing prototype.

Since we are discussing tiered storage in #3312 , and there are many talks about object storage solution. I'd like to ask if your proposal is appending segment/index file and then archiving them to S3 or writing to S3 directly?

The former is more like a tiered storage story and if you mean the latter, I'm worring about the performance since every write in iggy should be very small and you talked that S3 can't do appending.

GitHub link: https://github.com/apache/iggy/discussions/3232#discussioncomment-18372651

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org