Re: Tiered Storage for Iggy

["djanand (via GitHub)" <gi...@apache.org>]

GitHub user djanand added a comment to the discussion: Tiered Storage for Iggy

Reviving this now that VSR has landed, since that was the stated blocker. I'd like to help with it.

**Compression** isn't in the six areas, and I want to make the case that it belongs inside this design
### The Compression case
Tiering needs four things:
1) a background pass at segment seal
2) a stored format that stays readable across breaking version changes
3) a way to run multi-second CPU work without blocking the shard pump
4) and reads that resolve bytes which aren't raw local file.

Compression needs those same four, minus the network. Two things follow, namely
1) It doesn't strand work if tiering takes a while. A deployment on local disk with no interest in object storage still gets smaller segments, so the increment stands on its own.
2) And it isn't optional later. Over S3 the network is the bottleneck at roughly 90 MB/s per connection, so a 3x ratio is a 3x effective read, on top of lower egress and per-request cost and a 1 GiB upload dropping from ~11s to under 4.  _OpenDAL_ won't do it for us either: no compression layer, and S3 stores exactly the bytes you PUT. So it's work that has to happen regardless, and doing it first looks like the cheap ordering.

### Which point in the path
Three places it could go:

- _Ingest_, before consensus. Codec on every produce, and replicas receive compressed bytes.

- _Flush_, in `commit_messages_inner` when committed entries are written to the active segment. Produce stays clean, but the active segment holds compressed bytes, so reads of recent data decompress too.

- _Seal_, in `rotate_segment_at` once the segment is closed.

Seal is the only one that costs neither. The journal tail still serves Frozen buffers, the active segment is still read raw, and the decompress is confined to reads below the active segment's start offset.

### Where it lands in the six areas

**Migration** - Compress and upload are one pass over the same sealed segment rather than two

**Object storage I/O.** Flush units nest inside the 8 to 16 MiB objects cleanly, since a flush is at most size_of_messages_required_to_save (1 MiB default). One object holds many independently decompressible units, so range reads keep working.
I would argue that index should be stored local, and instead of object store. Size argues for local. An entry is `24 bytes` and there's one per flush, so a 1 GiB segment carries about `24 KiB` of index at the default cadence. Even a large tiered dataset's indexes fit on local disk comfortably. Writing them to object storage as well seems right for durability. Reading them from there on every cold poll is the part I'd want to avoid

**Runtime portability**

id |Approach | Cost
-- | -- | --
1 | Dedicated runtime + channel (@jayakasadev's) | A second runtime to own
2 | Stay on the shard, yield between flush units | Needs a fairness story, or compression starves reads
3 | OS thread outside the shard set | Breaks shared-nothing

_Option 2_ is what Seastar does. Rather than moving CPU work to another thread, it splits the work up and hands control back to the reactor between pieces: maybe_yield() checks a 500 µs quota and lets other tasks run ([tutorial](https://github.com/scylladb/seastar/blob/master/doc/tutorial.md)). Compressing one flush unit per slice would give natural break points. The catch is that yielding stops compression hogging the thread in one burst but doesn't cap its total CPU share. ScyllaDB handles that with scheduling groups, where compaction gets fewer shares than queries and so always loses.

### Proposed shape
Segment lifecycle. Compression and migration are two steps on one pipeline, and the index never leaves local disk:

```mermaid
flowchart LR
    A["Active segment<br/>raw, appendable"]
    B["Sealed<br/>raw"]
    C["Sealed<br/>compressed"]
    D[("Object storage")]

    A -->|"is_full at segment_size"| B
    B -->|"compress at seal"| C
    C -->|"migrate by age or size"| D

    IDX["Local index<br/>~24 KiB per GiB"]
    B -.-> IDX
    C -.-> IDX
    D -.-> IDX
```

Read path. Four resolutions, only two of which decompress, and the index lookup is local in every case:

```mermaid
flowchart TD
    P["poll offset N"] --> R{"resolve"}
    R -->|"journal tail"| J["serve Frozen<br/>zero-copy"]
    R -->|"active segment"| A["read raw from disk"]
    R -->|"sealed, local"| S["index lookup<br/>read unit<br/>decompress"]
    R -->|"sealed, in object storage"| O["index lookup<br/>ranged GET<br/>decompress"]
```




GitHub link: https://github.com/apache/iggy/discussions/3312#discussioncomment-18340122

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org