You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "samueldlightfoot (via GitHub)" <gi...@apache.org> on 2026/05/17 09:15:39 UTC
[PR] Cassandra 21134 - Direct I/O Background Writes [cassandra]
samueldlightfoot opened a new pull request, #4814:
URL: https://github.com/apache/cassandra/pull/4814
# CASSANDRA-21134: Direct I/O for background SSTable writes
## Summary
Adds an opt-in `O_DIRECT` write path for background SSTable producers, bypassing the OS
page cache for data that is unlikely to be re-read soon after being written. Memtable
flush data stays buffered (it is hot and benefits from the page cache).
Enabled via a new YAML knob:
```yaml
background_write_disk_access_mode: direct # default: standard
direct_write_buffer_size: 256KiB # aligned up to FS block size; auto-grows to chunk_length
```
The path is gated by:
1. config (`background_write_disk_access_mode == direct`),
2. table compression being enabled (required — uncompressed writers still use the buffered
path), and
3. an `OperationType`-keyed allowlist (`DataComponent#DIRECT_WRITE_SUPPORT`).
Selection happens centrally in `DataComponent.buildWriter`; producers are unchanged.
## Operations covered (DIO eligible)
| OperationType | Rationale |
| --------------------- | ------------------------------------------ |
| `COMPACTION` | append-only writer |
| `MAJOR_COMPACTION` | " |
| `TOMBSTONE_COMPACTION`| " |
| `ANTICOMPACTION` | " |
| `GARBAGE_COLLECT` | " |
| `CLEANUP` | " |
| `UPGRADE_SSTABLES` | " |
| `WRITE` | " |
| `STREAM` | chunked-receiver path (see ZCS exclusion) |
The allowlist is exhaustive: any new `OperationType` with `writesData == true` that is not
classified will fail static initialization (`AssertionError`).
## Operations NOT covered
| Path | Classification | Reason |
| ----------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------- |
| `FLUSH` (memtable flush) | `UNSUPPORTED_POLICY` | Just-flushed data is hot — keep it in the page cache. Memtable flushes always use buffered I/O. |
| `SCRUB` | `UNSUPPORTED_CORRECTNESS` | `tryAppend` needs `mark()` / `resetAndTruncate()`, which the DIO writer cannot satisfy. |
| Zero-Copy Streaming (ZCS) | n/a (path bypass) | Entire-SSTable streaming does not go through `DataComponent.buildWriter`; the DIO gate never runs. |
| Uncompressed writers | n/a (path bypass) | Only `CompressedSequentialWriter` has a DIO subclass in this change. |
Removing a `UNSUPPORTED_CORRECTNESS` entry requires code changes; removing
`UNSUPPORTED_POLICY` is a configuration / policy decision.
## Key code
- `io/DirectIoSupport.java` — eligibility enum (`SUPPORTED` / `UNSUPPORTED_CORRECTNESS` /
`UNSUPPORTED_POLICY` / `NOT_APPLICABLE`).
- `io/sstable/format/DataComponent.java` — central selection + allowlist + exhaustiveness
check; first activation per op is logged.
- `io/compress/DirectCompressedSequentialWriter.java` — new writer; aligned buffers, no
`mark()`/`resetAndTruncate()`.
- `io/compress/CompressedSequentialWriter.java` — refactored to allow the DIO subclass to
override the write chunk path; `writeChunk` contract documented and asserted.
- `config/Config.java`, `config/DatabaseDescriptor.java` — new knobs, validation, and
startup wiring; buffer size aligned to FS block size and auto-grown to chunk length.
- `service/StartupChecks.java` — fails fast if `direct` is requested on a platform/FS that
does not support `O_DIRECT`.
## Tests introduced
- `DirectCompressedSequentialWriterTest` (unit, 818 lines) — covers the DIO writer in
isolation: chunk-boundary alignment, buffer auto-expansion to chunk length, abort/close
paths, checksum + compression-info component correctness, error handling.
- `DataComponentDirectWriteSelectionTest` (unit) — verifies the selection matrix:
per-OperationType eligibility, exhaustiveness assertion, compression-enabled gate,
config-mode gate.
- `StreamingDirectWriteTest` (in-JVM distributed) — proves chunked streaming
(`CassandraStreamReader` / `CassandraCompressedStreamReader` →
`BigTableWriter.openDataWriter` → `OperationType.STREAM`) selects the DIO writer when
enabled; ZCS is disabled in the test since it bypasses the selection point.
- `DirectIoTestUtils` — shared helpers (FS block size, alignment) for the suites above.
- `AntiCompactionTest`, `CompactionsTest` — extended to exercise the DIO path end-to-end
for the compaction operations in the allowlist.
- `DatabaseDescriptorTest` — validation of the new knobs (mode parsing, buffer-size
alignment, defaults).
## Not in scope
- Direct I/O on the read path.
- Uncompressed SSTable writers.
- ZCS streaming.
- Memtable flush.
patch by Sam Lightfoot; reviewed by <Reviewers> for CASSANDRA21134
```
The [Cassandra Jira](https://issues.apache.org/jira/browse/CASSANDRA-21134)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 21134 - Direct I/O Background Writes [cassandra]
Posted by "samueldlightfoot (via GitHub)" <gi...@apache.org>.
samueldlightfoot closed pull request #4814: Cassandra 21134 - Direct I/O Background Writes
URL: https://github.com/apache/cassandra/pull/4814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org