You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@iggy.apache.org by "djanand (via GitHub)" <gi...@apache.org> on 2026/10/05 06:47:45 UTC

Design proposal: bound segment time span ( `segment_ms` )

GitHub user djanand created a discussion: Design proposal: bound segment time span ( `segment_ms` )

### Goal
Put a ceiling on how much time a segment spans, so retention works on partitions that do not fill a segment quickly, and so message_expiry can be honored with bounded error

### Issue
Segments seal on size alone. `Segment::is_full()` compares against the per-topic segment_size, 1 GiB by default, and there is no time trigger. Retention currently looks at sealed segments. `Segment::is_expired()` and bails early on anything unsealed. So essentially,
* Expiration : `message_expiry` runs on message timestamps, continuously regardless of segment state.
* Eligibility : sealing decides if a segment is eligible for expiration.

The expiry condition can be true for weeks without sealing the segment.

### What it looks like in practice
With `segment_size = 1 GiB` and `message_expiry = 30 days` :
1. A topic at 1 MB/hour takes about ~45 days ( six weeks) to fill a segment. For those six weeks nothing seals, so nothing expires. The operator asked for 30 days and gets six weeks.
2. A topic at 1 MB/s fills a segment in roughly 18 minutes, seals constantly, and behaves exactly as documented. Which is probably why this has not come up.
3. A _decommissioned topic_ stops leaves every partition sitting on an unsealed segment, maybe 500 MB each, none reaching 1 GiB and none ever sealing. The same thing happens inside a live topic whenever traffic skews: a partition that stops receiving keys is idle whether or not the topic is.

### How late deletion runs
Expiration compates `max_timestamp`, the newest message in a segment, so nothing is deleted until the newest message is aged out. The oldest message in a segment lives as long as the segment spans. For e.g taking `message_expiry = 1h`

#### Current state
| Throughput | 1 GiB spans | oldest data lives |
|--------|--------|--------|
| 1 MB/s | ~18 minutes | ~1h 18m |
| 12 KB/s | ~24 hours | ~25 hours |
| 1 KB/s | ~12 days | ~12 days |

#### Proposed State (with segment_ms)
Setting segment_ms = 1h caps the span, and the worst case stops depending on throughput:

Worst case goes from message_expiry + however long a segment takes to fill to message_expiry + segment_ms, which is the guarantee Kafka gives with segment.ms.

### High-level shape
* A per-topic `segment_ms` alongside `segment_size`, parsed the same way. Rotation becomes size-or-span.
```
segment_ms = "none" size-only, today's behaviour
segment_ms = "default" take the server default (7days) - [refer Open Questions (Q2)]
segment_ms = "7 days" bound the span at this
```

* _Keyed on replicated message timestamps, not wall clock_ - Segment boundaries land at the same offset on every replica today, and that is deliberate. The rollover is keyed on accumulated bytes so flush grouping cannot shift it.
* Compare arriving batch against the segment i.e.
```
roll if segment.start_timestamp != 0
&& incoming.base_timestamp - segment.start_timestamp >= segment_ms
```
Both values are primary-assigned and replicated, so every replica rolls at the same message and boundaries stay aligned. Additionally, the roll has to happen before the arriving batch is appended. If it lands in the old segment first, that segment's `max_timestamp` jumps forward and its contents stop being expirable for another full `message_expiry`

#### Preallocated segments need trimming at seal
* `preallocate_segments` reserves exactly `segment_size` up front. That is safe today because a segment only seals once it has reached or exceeded that size, so the reservation is always fully used and there is nothing to reclaim.
* A slow topic with preallocation on would reserve 1 GiB, write a few MB into it, and roll. fallocate is called with FALLOC_FL_KEEP_SIZE, so those blocks stay allocated to the file, and nothing in the seal path truncates. The file reports a few MB to ls and holds 1 GiB on disk until retention deletes it, which could be the whole retention window away
* `max_topic_size` compares against `segment.size`, which counts written bytes. It would see a few MB while the disk fills
* So the proposal needs a trim at seal, reducing the file to its written length. Kafka does this in `onBecomeInactiveSegment`, which calls `log.trim()`

### What this does not cover

* A partition that stops receiving writes entirely. Stream time only advances when messages arrive, so the decommissioned case above still never seals.
* Catching this needs actual wall-clock time, and no replica can read a _clock_ without drifting from its peers. So it is either 1) a timer per node, accepting the divergence, or 2) the primary decides and replicates the rollover. The second seems right to me, but it means a new `Operation`, and `router.rs` says the version fence that would make that safe during a rolling upgrade is not there yet. Both are bigger than this change, so I would keep them separate.

### Open Questions
1. _Mutable, not create-only_, unless someone sees a reason I have missed. `segment_size` is fixed at creation because `max_topic_size` admission uses it to work out the smallest cap it can enforce, which only holds if existing segments really are that size. No calculation uses a time setting that way. Mutable matters here anyway: the topics with this problem already exist, and create-only would mean recreating them to fix it.
2. _Server default_. Filling 1 GiB in 7 days works out at about 1.8 KB/s, so a 7 day default would only ever affect partitions writing slower than that. Anything faster fills a segment inside the week and still rotates on size, exactly as it does now. Turning it on by default therefore cannot disturb a healthy topic, and the only partitions it touches are the ones that need it already. Inclined to have default as on ( _in the near future_ ), rather than `none`
3. How the seal-time trim gets done. - Going through compio submits` IORING_OP_FTRUNCATE`, which only exists from _kernel 6.9_, and on anything older the driver reaches for a blocking pool the shard does not have and panics. Proposal is copy `FileStorage::truncate` rather than invent something, but I would rather hear it is the house pattern than assume. It would make a failed trim non-fatal, a warning and carry on, since the segment is perfectly good either way and you have only failed to give some blocks back.

### Affected area / component
* Iggy server
* Configuration

### Alternatives considered
1. _A wall-clock timer per partition_. Covers the idle case, which stream time does not, and breaks byte-identical segments across replicas. Rejected because we 1) loose ability to compare cross replicas (i.e checksum ) 2) Tiering gets harder 3) Its a one-way door and everything has to tolerate divergence
2. _Lower `segment_size` at creation_. Only helps new topics, since the key is _create-only_, and expresses the wrong intent by trading segment count against a size budget when the requirement is about time.
3. Idle topic. But it needs a new consensus Operation, and a node that has not been upgraded wedges its whole group silently on meeting one, with no version fence yet to stop that. Bigger than this change, probably its own issue

### What I'd like from this discussion

* Confirmation that retention skipping unsealed segments is a gap rather than a deliberate trade
* Whether I have the replica-boundary constraint right. The whole design is shaped around keeping segment boundaries identical, and if that turns out to be softer than I think, a wall-clock timer is simpler and covers the idle case too
* A steer on the server default, since it is the one decision that changes behaviour for deployments that exist today
* Whether `FileStorage::truncate` is the pattern to copy for the seal-time trim, or there is a better one
* Whether the uncovered idle-partition case wants its own issue now, or can sit documented until someone hits it

### Contribution
[x] I'm willing to submit a pull request to implement this feature

### Good first issue
[ ] unchecked - touches persistence and the rotation path

GitHub link: https://github.com/apache/iggy/discussions/4402

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: bound segment time span ( `segment_ms` )

Posted by "hubcio (via GitHub)" <gi...@apache.org>.

GitHub user hubcio added a comment to the discussion: Design proposal: bound segment time span ( `segment_ms` )

i skimmed thru this proposal and I like the idea. i will review this in detail this week.

GitHub link: https://github.com/apache/iggy/discussions/4402#discussioncomment-18757103

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: bound segment time span ( `segment_ms` )

Posted by "djanand (via GitHub)" <gi...@apache.org>.

GitHub user djanand added a comment to the discussion: Design proposal: bound segment time span ( `segment_ms` )

Thanks, these are really useful details and some that I overlooked. Still working through it.

GitHub link: https://github.com/apache/iggy/discussions/4402#discussioncomment-19117059

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: bound segment time span ( `segment_ms` )

Posted by "hubcio (via GitHub)" <gi...@apache.org>.

GitHub user hubcio added a comment to the discussion: Design proposal: bound segment time span ( `segment_ms` )

you better do that after #4561 is merged ;)

GitHub link: https://github.com/apache/iggy/discussions/4402#discussioncomment-19119104

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: bound segment time span ( `segment_ms` )

Posted by "hubcio (via GitHub)" <gi...@apache.org>.

GitHub user hubcio added a comment to the discussion: Design proposal: bound segment time span ( `segment_ms` )

@djanand overall i like this. a few things need to be addressed:

1. expiry works on whole segments, not messages. the cleaner only removes sealed segments, and only once `max_timestamp + message_expiry <= now`. active-segment protection is deliberate and tested, so i'd keep it and add the missing rotation trigger. with your numbers (45-day fill, 30-day expiry) the oldest messages become eligible at ~75 days, not six weeks. ([expiry][expiry])

2. `message_expiry + segment_ms` isn't a hard bound. you already scope out idle partitions, but sparse ones hit the same wall: rotation is only checked on append, so the roll waits for the next batch. cleanup ticks and persistence checkpoints add delay. consumer and consumer-group offsets also block removal, with no timeout (external groups excepted). a stopped consumer pins expired segments past its offset until that offset or its group is deleted. i'd frame this as finer retention granularity, not a max message lifetime, and open the idle-partition issue now. ([deletion gates][gates], [offsets][barrier])

3. the WAL and materialization have to make the same call. `SegmentState::reserve` picks the physical segment before the batch is written, and materialization checks that placement. if only the logical rotation looks at time, that check rejects the commit, the partition gets fenced and the node shuts down. the decision has to be per batch, independent of flush grouping, and reproducible after restart, rollback and state transfer. `SegmentState` has no timestamp today. the open segment's first timestamp has to be persisted with it, or recovered from the segment wherever the WAL rebuilds its tail. segments also only seal during materialization (by default every 1024 messages or 1 MiB), so a time roll has to make materialization due too. ([WAL placement][reserve], [placement check][prefix], [fatal commit][fatal], [flush trigger][flush])

4. if it's mutable, the design needs to say which batch first uses a new value and how replay gets the same answer. the runtime knobs are create-only because nothing re-pushes them to a live partition and they describe the storage layout. `max_topic_size` admission depends on that, it isn't the cause. `message_expiry` and `max_topic_size` can be mutable because only the local, unreplicated cleaner enforces them, rereading metadata each pass. `segment_ms` feeds replicated WAL placement instead, and the WAL already refuses a changed segment size. topic updates also commit in the metadata consensus group, not the partition's, so replicas can pick up a new threshold at different batches. re the version fence: since #4387 (merged after you posted), the replica handshake rejects peers from a different release, so an old replica never sees a new `Operation`. ([runtime knobs][knobs], [cleaner][cleaner], [WAL size check][walsize], [build identity][identity])

5. for the trim, call `server_common::fs_utils::truncate_file`. it avoids `IORING_OP_FTRUNCATE`, and since #4311 `FileStorage::truncate` delegates to it, so there's nothing to copy. the length is harder: the public segment and the retained WAL path are hard links to one inode, and WAL writes can run ahead of materialization. trim only after the WAL tail has left the segment, and only to the length where WAL placement, `segment.size` and the file length agree. anything shorter deletes bodies the WAL still references. a failed trim can be nonfatal, as long as that doesn't weaken persistence or recovery error handling. ([helper][helper], [shared inode][inode], [WAL ahead][ahead])

for the default, i'd go opt-in. low throughput can be normal, and this changes rotation, allocation and retention for those topics. topic defaults are catalog constants, not server settings, and admission persists them on new topics. so please spell out separately what new topics get and what a missing key means for existing ones. ([defaults][defaults])

it also needs a lower bound. `MIN_TOPIC_SEGMENT_SIZE` exists because too many segments make a partition unservable for state transfer (65536 manifest entries max), and a time trigger skips that floor. ([floor][floor])

tests i'd want to see:

- sparse arrivals, including a time roll below the flush thresholds
- the threshold-crossing batch landing entirely in the new segment
- identical boundaries under different flush grouping
- recovery with WAL writes ahead of materialization
- setting changes mid-write
- preallocation cleanup that checks allocated blocks and message contents

i think PR for this will be huge 🥲

[expiry]: https://github.com/apache/iggy/blob/fa86c2998/core/partitions/src/segment.rs#L73-L87
[gates]: https://github.com/apache/iggy/blob/fa86c2998/core/partitions/src/iggy_partition.rs#L10852-L10879
[barrier]: https://github.com/apache/iggy/blob/fa86c2998/core/partitions/src/iggy_partition.rs#L10760-L10763
[reserve]: https://github.com/apache/iggy/blob/fa86c2998/core/journal/src/partition_journal/segments.rs#L764-L797
[prefix]: https://github.com/apache/iggy/blob/fa86c2998/core/partitions/src/persistence.rs#L700-L738
[fatal]: https://github.com/apache/iggy/blob/fa86c2998/core/server/src/boot/threads.rs#L776-L795
[flush]: https://github.com/apache/iggy/blob/fa86c2998/core/partitions/src/iggy_partition.rs#L8684-L8693
[knobs]: https://github.com/apache/iggy/blob/fa86c2998/core/common/src/types/options/mod.rs#L459-L478
[cleaner]: https://github.com/apache/iggy/blob/fa86c2998/core/server/src/segment_cleaner.rs#L18-L27
[walsize]: https://github.com/apache/iggy/blob/fa86c2998/core/journal/src/partition_journal/segments.rs#L75-L78
[identity]: https://github.com/apache/iggy/blob/fa86c2998/core/message_bus/src/replica/handshake.rs#L534-L544
[helper]: https://github.com/apache/iggy/blob/fa86c2998/core/server_common/src/fs_utils.rs#L170-L210
[inode]: https://github.com/apache/iggy/blob/fa86c2998/core/journal/src/partition_journal/segments.rs#L670-L724
[ahead]: https://github.com/apache/iggy/blob/fa86c2998/core/partitions/src/iggy_partition.rs#L8952-L8953
[defaults]: https://github.com/apache/iggy/blob/fa86c2998/core/common/src/types/options/mod.rs#L234-L238
[floor]: https://github.com/apache/iggy/blob/fa86c2998/core/common/src/types/options/mod.rs#L291-L295


GitHub link: https://github.com/apache/iggy/discussions/4402#discussioncomment-18899478

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: bound segment time span ( `segment_ms` )

Posted by "djanand (via GitHub)" <gi...@apache.org>.

GitHub user djanand edited a discussion: Design proposal: bound segment time span ( `segment_ms` )

### Goal
Put a ceiling on how much time a segment spans, so retention works on partitions that do not fill a segment quickly, and so message_expiry can be honored with bounded error

Related to #3312. A partition that never seals cannot be compressed or offloaded either, so bounding the span can be a prerequisite for the tiering work discussed there rather than a parallel concern.

The expiry condition can be true for weeks without sealing the segment.

#### Proposed State (with segment_ms)
Setting segment_ms = 1h caps the span, and the worst case stops depending on throughput:

Worst case goes from message_expiry + however long a segment takes to fill to message_expiry + segment_ms, which is the guarantee Kafka gives with segment.ms.

### What this does not cover

### Affected area / component
* Iggy server
* Configuration

### What I'd like from this discussion

### Contribution
[x] I'm willing to submit a pull request to implement this feature

### Good first issue
[ ] unchecked - touches persistence and the rotation path

GitHub link: https://github.com/apache/iggy/discussions/4402

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org

Re: Design proposal: bound segment time span ( `segment_ms` )

Posted by "djanand (via GitHub)" <gi...@apache.org>.

GitHub user djanand added a comment to the discussion: Design proposal: bound segment time span ( `segment_ms` )

thanks for looking into this @hubcio.

GitHub link: https://github.com/apache/iggy/discussions/4402#discussioncomment-18763486

----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org