You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "andreinnanu (via GitHub)" <gi...@apache.org> on 2026/10/11 18:20:29 UTC
[PR] feat(server): disk usage accounting [iggy]
andreinnanu opened a new pull request, #4488:
URL: https://github.com/apache/iggy/pull/4488
## Which issue does this PR address?
Relates to #4140
This PR alone doesn't solve the issue. A following one will.
## Rationale
Enables accounting disk usage, preparing the ground for the configurable disk usage limit discussed within #4140.
As agreed there, this is the first of two PRs: accounting here, the limit and the refusal of new messages in the follow-up.
## What changed?
Iggy had no view of how full its data volume is, so the disk usage limit planned in #4140 had nothing to check against.
A periodic `statvfs` sample alone is not enough: a burst can fill the remaining space between two samples.
A process-wide estimate (`DISK_USAGE`) of the bytes used on the data volume was added.
The write sites charge what they allocate, and the existing `iggy-system-stats` sampler seeds the estimate at boot and re-bases it from `statvfs`, so writes on the data volume which are not under Iggy's control are seen with every sample.
Preallocation charges only newly allocated space: re-reserving an existing segment at recovery does not change the estimate.
The estimate is exported as the `disk_used_bytes` gauge on `/metrics`. Accounting only: nothing is refused yet.
### Open question based on #4140 discussion
How do we want to approach freezing the sampler?
The goal was to add a burst test with the sampler frozen, but currently there is no mechanism for that. Reusing `sysinfo_print_interval` doesn't work because its minimum value is 1s.
## Local Execution
- Passed: /metrics includes the `disk_used_bytes` gauge and its value and follows `df` while a producer writes messages
- Pre-commit hooks pass
<!--
You must run your code locally before submitting.
"Relying on CI" is not acceptable - PRs from authors who haven't run the code will be closed.
Did you have `prek` installed? It runs automatically on commit and covers all project languages. See [CONTRIBUTING.md](https://github.com/apache/iggy/blob/master/CONTRIBUTING.md).
-->
## AI Usage
- used Claude Code for generating tests & some of the functions
- verified by manual tests & code review
<!--
If AI tools were used, please answer:
1. Which tools? (e.g., GitHub Copilot, Claude, ChatGPT)
2. Scope of usage? (e.g., autocomplete, generated functions, entire implementation)
3. How did you verify the generated code works correctly?
4. Can you explain every line of the code if asked?
If no AI tools were used, write "None" or delete this section.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4466372066
##########
core/journal/src/durable_storage.rs:
##########
@@ -65,6 +66,14 @@ pub struct StorageEntry {
pub trait DurableStorage {
type File: DurableFile;
+ /// Count `bytes` about to be allocated on disk towards the process-wide
+ /// disk usage estimate. Hold the returned charge until the write finishes
+ /// or fails. Backends that do not write to the data volume ignore it.
+ #[must_use = "a charge settles when dropped, so hold it until the write finishes"]
+ fn charge_allocation(&self, _bytes: u64) -> Option<Charge<'static>> {
Review Comment:
warning: this no-op default lets a new wrapper over `DiskStorage` drop every charge with no compile error - `SimStorage` and three test wrappers already inherit it. drop the method and call `DISK_USAGE.charge` at the eight call sites, like `file_storage.rs` does.
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
warning: btrfs and openzfs move `f_bfree` only at the next commit (30 s on btrfs), so after `rebase` drops settled charges the gauge reads low until then. keep settled charges past a commit on those filesystems, or document the ext4/xfs assumption.
##########
core/partitions/src/messages_writer.rs:
##########
@@ -100,6 +104,7 @@ impl MessagesWriter {
file,
messages_size_bytes,
fsync,
+ reserved_end: preallocate_size.map_or(0, |size| size.as_bytes_u64()),
Review Comment:
warning: `reserved_end` assumes the reservation worked, but `preallocate_file` only logs a failed or unsupported `fallocate` (and openzfs returns ok without allocating), so writes below it charge nothing. return the bytes actually reserved, measured by `st_blocks` growth, and set `reserved_end` from that.
also at `core/journal/src/partition_journal/segments.rs:464`.
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
Review Comment:
nit: all shards bump the same two global atomics on every charge, so these cache lines bounce between cores. profile a many-shard small-batch run before deciding whether per-shard slots are needed.
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
Review Comment:
nit: before the first sample this returns just the charged total, so every reader has to remember `is_measured()`. return `Option<u64>` and drop the public `is_measured`.
##########
core/server/src/http/metrics.rs:
##########
@@ -158,6 +159,55 @@ fn encode_topic_gauge(
Ok(())
}
+/// Exports the disk usage estimate as a gauge, read at encode time like
+/// [`StatsRollupUnderflows`].
+///
+/// The value is the last `statvfs` sample of the volume holding the data
+/// directory plus what Iggy has written since, so it moves with a burst
+/// instead of once a second. No series is exported until the first sample,
+/// since the count alone says nothing about the volume.
+///
+/// It holds the estimate it reads, so a test can export one of its own
+/// instead of the process-wide one other tests charge concurrently.
+#[derive(Debug)]
+struct DiskUsedBytes(&'static DiskUsage);
+
+impl Collector for DiskUsedBytes {
+ fn encode(&self, mut encoder: DescriptorEncoder) -> Result<(), std::fmt::Error> {
+ if !self.0.is_measured() {
+ return Ok(());
+ }
+ let gauge = ConstGauge::new(gauge_value(self.0.used_bytes()));
+ let metric_encoder = encoder.encode_descriptor(
+ "disk_used_bytes",
+ "estimated bytes used on the volume holding the data directory",
+ None,
+ gauge.metric_type(),
+ )?;
+ gauge.encode(metric_encoder)
+ }
+}
+
+/// Exports the bytes the write sites have charged since the process started,
+/// the part of [`DiskUsedBytes`] that samples never move. Its increase over an
Review Comment:
nit: the doc says the increase is what iggy allocated, but failed writes stay charged and a failed or unsupported reservation adds the whole segment at once. say that here.
##########
core/server_common/src/fs_utils.rs:
##########
@@ -93,6 +108,36 @@ pub async fn preallocate_file(file: &fs::File, file_path: &Path, len: u64) {
}
}
+/// Reserves `len` bytes and charges `usage` for the blocks it allocates.
+/// Recovery reserves segments that already own their blocks again, which
+/// allocates nothing and must not be counted as disk usage.
+#[cfg(target_os = "linux")]
+fn reserve(file: &std::fs::File, len: i64, usage: &DiskUsage) -> io::Result<()> {
+ // `st_blocks` counts 512-byte units whatever the filesystem's block size.
+ let allocated = || file.metadata().ok().map(|metadata| metadata.blocks() * 512);
Review Comment:
nit: bare `512` for the `st_blocks` unit - give it a named constant in each crate, like `FILE_BLOCK_BYTES` in the journal tests.
also at line 322 and `core/integration/src/harness/disk.rs:137`.
##########
core/server/src/http/metrics.rs:
##########
@@ -158,6 +159,55 @@ fn encode_topic_gauge(
Ok(())
}
+/// Exports the disk usage estimate as a gauge, read at encode time like
+/// [`StatsRollupUnderflows`].
+///
+/// The value is the last `statvfs` sample of the volume holding the data
+/// directory plus what Iggy has written since, so it moves with a burst
+/// instead of once a second. No series is exported until the first sample,
+/// since the count alone says nothing about the volume.
+///
+/// It holds the estimate it reads, so a test can export one of its own
+/// instead of the process-wide one other tests charge concurrently.
+#[derive(Debug)]
+struct DiskUsedBytes(&'static DiskUsage);
+
+impl Collector for DiskUsedBytes {
+ fn encode(&self, mut encoder: DescriptorEncoder) -> Result<(), std::fmt::Error> {
+ if !self.0.is_measured() {
+ return Ok(());
+ }
+ let gauge = ConstGauge::new(gauge_value(self.0.used_bytes()));
+ let metric_encoder = encoder.encode_descriptor(
+ "disk_used_bytes",
+ "estimated bytes used on the volume holding the data directory",
+ None,
+ gauge.metric_type(),
+ )?;
+ gauge.encode(metric_encoder)
+ }
+}
+
+/// Exports the bytes the write sites have charged since the process started,
+/// the part of [`DiskUsedBytes`] that samples never move. Its increase over an
+/// interval is what Iggy allocated in it; the gauge also moves with deletes
+/// and with other processes on the volume.
+#[derive(Debug)]
+struct DiskChargedBytes(&'static DiskUsage);
+
+impl Collector for DiskChargedBytes {
+ fn encode(&self, mut encoder: DescriptorEncoder) -> Result<(), std::fmt::Error> {
+ let counter = ConstCounter::new(self.0.charged_bytes());
+ let metric_encoder = encoder.encode_descriptor(
+ "disk_charged_bytes",
Review Comment:
nit: `disk_charged_bytes` leaks an internal accounting word into a public metric name, and the PR body only mentions `disk_used_bytes`. pick the operator-facing name now, since renaming a metric later breaks dashboards, and list it in the body.
##########
core/integration/tests/server/disk_usage.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The disk usage accounting of a running server.
+//!
+//! The burst tests compare what the write sites charged (`disk_charged_bytes`,
+//! which samples never move) with what the data directory allocated over the
+//! same burst. Both belong to this server alone, so the sampler keeps running
+//! and the tests may run beside others.
+//!
+//! The foreign file test compares the `disk_used_bytes` estimate with the
+//! whole volume, so it must run alone: a `#[serial]` excludes the other tests
+//! under `cargo test`, and `.config/nextest.toml` gives it every thread under
+//! nextest.
+
+use std::path::Path;
+use std::time::{Duration, Instant};
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::harness::disk::{allocated_bytes, volume_used_bytes};
+use integration::harness::seeds::{self, names};
+use serial_test::{parallel, serial};
+use tokio::time::sleep;
+
+use super::http_client::HttpClient;
+
+const KIB: u64 = 1024;
+const MIB: u64 = 1024 * KIB;
+const BURST_TOPIC: &str = "burst";
+
+/// Small enough that the burst fills four segments and opens a fifth, so
+/// reservations, the batch that crosses each segment end and a reservation
+/// still mostly unwritten are all part of it.
+const SEGMENT_BYTES: u64 = 4 * MIB;
+const MESSAGE_BYTES: usize = 64 * KIB as usize;
+const MESSAGES_PER_BATCH: usize = 4;
+
+/// 18 MiB of bodies in 72 prepares. Every prepare adds a 4 KiB WAL record, so
Review Comment:
nit: the checkpoint trigger counts whole padded prepares, not 4 KiB WAL records, so this burst is about 18.3 MiB against a 128 MiB trigger. the conclusion holds, only the math in this comment is off.
##########
core/server/src/sysinfo_probe.rs:
##########
@@ -138,9 +139,15 @@ impl SystemStats {
fn capture(sys: &mut SysinfoSystem, previous_open_files: u64, scan_open_files: bool) -> Self {
let host = HOST_IDENTITY.get_or_init(HostIdentity::probe);
let probe = SystemProbe::capture(sys);
- let disk = STATS_DATA_PATH
- .get()
- .and_then(|path| fs2::statvfs(path).ok());
+ // Each sample also re-bases the disk usage estimate the write sites
+ // charge, and the first one, taken before the shards start, seeds it.
+ let disk = STATS_DATA_PATH.get().and_then(|path| {
+ DISK_USAGE.rebase(|| {
+ let stats = fs2::statvfs(path).ok()?;
+ let used_bytes = stats.total_space().saturating_sub(stats.free_space());
Review Comment:
nit: this matches `df` used (`f_blocks - f_bfree`), while `/stats` free uses `f_bavail`, so the two disagree by the root reserve. keep the formula and say in the gauge help text that used means total minus free blocks, like `df`.
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
Review Comment:
nit: both `base` accesses are relaxed, so on aarch64 a reader can pair a new `base` with a stale `charged` and read below the sample. store `base` with release in `rebase`, load it with acquire here, and keep this load order.
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -332,6 +332,9 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
.storage
.open(&reference.path(&self.directory), OpenMode::Create)
.await?;
+ let _charge = self
Review Comment:
nit: `_charge` lives across the `sync().await` below, so the bytes settle only after the fsync and a sample during it counts them twice until the next sample. drop the charge right after the write.
##########
core/integration/tests/server/disk_usage.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The disk usage accounting of a running server.
+//!
+//! The burst tests compare what the write sites charged (`disk_charged_bytes`,
+//! which samples never move) with what the data directory allocated over the
+//! same burst. Both belong to this server alone, so the sampler keeps running
+//! and the tests may run beside others.
+//!
+//! The foreign file test compares the `disk_used_bytes` estimate with the
+//! whole volume, so it must run alone: a `#[serial]` excludes the other tests
+//! under `cargo test`, and `.config/nextest.toml` gives it every thread under
+//! nextest.
+
+use std::path::Path;
+use std::time::{Duration, Instant};
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::harness::disk::{allocated_bytes, volume_used_bytes};
+use integration::harness::seeds::{self, names};
+use serial_test::{parallel, serial};
+use tokio::time::sleep;
+
+use super::http_client::HttpClient;
+
+const KIB: u64 = 1024;
+const MIB: u64 = 1024 * KIB;
+const BURST_TOPIC: &str = "burst";
+
+/// Small enough that the burst fills four segments and opens a fifth, so
+/// reservations, the batch that crosses each segment end and a reservation
+/// still mostly unwritten are all part of it.
+const SEGMENT_BYTES: u64 = 4 * MIB;
+const MESSAGE_BYTES: usize = 64 * KIB as usize;
+const MESSAGES_PER_BATCH: usize = 4;
+
+/// 18 MiB of bodies in 72 prepares. Every prepare adds a 4 KiB WAL record, so
+/// the WAL grows by 288 KiB, and the WAL checkpoint (half the 256 MiB budget)
+/// stays out of reach: the counter never sees the frees a checkpoint makes.
+const BATCHES: usize = 72;
+
+/// What the write sites rightly leave out: the end of every segment and index
+/// file rounded up to a filesystem block, and the server's own log lines.
+/// Below the burst's 288 KiB of WAL records, so a missing WAL charge fails.
+const BURST_BOUND: u64 = 128 * KIB;
+
+/// Quiet scrapes in a row before the charges count as settled. Sends return
+/// once committed, which can be before the WAL writer and materialization
+/// have written what they charge.
+const SETTLED_SCRAPES: usize = 5;
+const SETTLE_DEADLINE: Duration = Duration::from_secs(10);
+
+/// What else moves the volume between a sample and the snapshot after it
+/// (other processes, the server's own log lines), plus block rounding.
+const TOLERANCE: u64 = MIB;
+const FOREIGN_FILE_BYTES: usize = 16 * MIB as usize;
+/// Comfortably more than one sampling interval (1 s).
+const SAMPLE_DEADLINE: Duration = Duration::from_secs(5);
+const POLL_INTERVAL: Duration = Duration::from_millis(100);
+
+#[tokio::test]
+#[parallel]
+async fn given_a_burst_when_the_charges_settle_should_match_the_data_directory_growth() {
+ assert_burst_is_charged_with_preallocation(false).await;
+}
+
+#[tokio::test]
+#[parallel]
+async fn given_a_burst_into_preallocated_segments_when_the_charges_settle_should_match_the_data_directory_growth()
Review Comment:
nit: this test assumes `fallocate` with `FALLOC_FL_KEEP_SIZE` works on the data volume - where it doesn't, each segment charges a phantom reservation and the diff blows past `BURST_BOUND`. probe it on the data path first and skip unless `st_blocks` grow.
##########
core/integration/tests/server/disk_usage.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The disk usage accounting of a running server.
+//!
+//! The burst tests compare what the write sites charged (`disk_charged_bytes`,
+//! which samples never move) with what the data directory allocated over the
+//! same burst. Both belong to this server alone, so the sampler keeps running
+//! and the tests may run beside others.
+//!
+//! The foreign file test compares the `disk_used_bytes` estimate with the
+//! whole volume, so it must run alone: a `#[serial]` excludes the other tests
Review Comment:
nit: `#[serial]` only excludes other `#[serial]`/`#[parallel]` tests, so unmarked server tests like the `external_auth.rs` ones still run beside this under `cargo test`. say only nextest isolates it, or mark those tests `#[parallel]`.
##########
core/partitions/src/iggy_index_writer.rs:
##########
@@ -130,11 +131,10 @@ impl IggyIndexWriter {
let len = indexes.len();
let file = &self.file;
- (&*file)
- .write_all_at(indexes, position)
- .await
- .0
- .map_err(|_| IggyError::CannotSaveIndexToSegment)?;
+ let charge = DISK_USAGE.charge(len as u64);
Review Comment:
simplification: nothing awaits between `drop(charge)` and the end of the scope, so a plain `let _charge = ...` settles at the same point. use that and keep the original write chain.
also at `messages_writer.rs:203`, `file_storage.rs:130`, `prepare_journal.rs:963`, `partition_journal.rs:863`, `:1395`, `:1470` and `offset_storage.rs:374`, `:418`.
##########
core/integration/tests/server/disk_usage.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The disk usage accounting of a running server.
+//!
+//! The burst tests compare what the write sites charged (`disk_charged_bytes`,
+//! which samples never move) with what the data directory allocated over the
+//! same burst. Both belong to this server alone, so the sampler keeps running
+//! and the tests may run beside others.
+//!
+//! The foreign file test compares the `disk_used_bytes` estimate with the
+//! whole volume, so it must run alone: a `#[serial]` excludes the other tests
+//! under `cargo test`, and `.config/nextest.toml` gives it every thread under
+//! nextest.
+
+use std::path::Path;
+use std::time::{Duration, Instant};
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::harness::disk::{allocated_bytes, volume_used_bytes};
+use integration::harness::seeds::{self, names};
+use serial_test::{parallel, serial};
+use tokio::time::sleep;
+
+use super::http_client::HttpClient;
+
+const KIB: u64 = 1024;
+const MIB: u64 = 1024 * KIB;
+const BURST_TOPIC: &str = "burst";
+
+/// Small enough that the burst fills four segments and opens a fifth, so
+/// reservations, the batch that crosses each segment end and a reservation
+/// still mostly unwritten are all part of it.
+const SEGMENT_BYTES: u64 = 4 * MIB;
+const MESSAGE_BYTES: usize = 64 * KIB as usize;
+const MESSAGES_PER_BATCH: usize = 4;
+
+/// 18 MiB of bodies in 72 prepares. Every prepare adds a 4 KiB WAL record, so
+/// the WAL grows by 288 KiB, and the WAL checkpoint (half the 256 MiB budget)
+/// stays out of reach: the counter never sees the frees a checkpoint makes.
+const BATCHES: usize = 72;
+
+/// What the write sites rightly leave out: the end of every segment and index
+/// file rounded up to a filesystem block, and the server's own log lines.
+/// Below the burst's 288 KiB of WAL records, so a missing WAL charge fails.
+const BURST_BOUND: u64 = 128 * KIB;
+
+/// Quiet scrapes in a row before the charges count as settled. Sends return
+/// once committed, which can be before the WAL writer and materialization
+/// have written what they charge.
+const SETTLED_SCRAPES: usize = 5;
+const SETTLE_DEADLINE: Duration = Duration::from_secs(10);
+
+/// What else moves the volume between a sample and the snapshot after it
+/// (other processes, the server's own log lines), plus block rounding.
+const TOLERANCE: u64 = MIB;
+const FOREIGN_FILE_BYTES: usize = 16 * MIB as usize;
+/// Comfortably more than one sampling interval (1 s).
+const SAMPLE_DEADLINE: Duration = Duration::from_secs(5);
+const POLL_INTERVAL: Duration = Duration::from_millis(100);
+
+#[tokio::test]
+#[parallel]
+async fn given_a_burst_when_the_charges_settle_should_match_the_data_directory_growth() {
+ assert_burst_is_charged_with_preallocation(false).await;
+}
+
+#[tokio::test]
+#[parallel]
+async fn given_a_burst_into_preallocated_segments_when_the_charges_settle_should_match_the_data_directory_growth()
+ {
+ assert_burst_is_charged_with_preallocation(true).await;
+}
+
+#[tokio::test]
+#[serial]
+async fn given_a_file_written_outside_iggy_when_the_next_sample_runs_should_count_it_until_deleted()
+{
+ let harness = start_server().await;
+ let http = HttpClient::login_root(&harness).await;
+ let data_path = harness.server().data_path();
+ // Beside the data directory, so on the same volume but never charged.
+ let foreign_path = data_path.with_extension("foreign");
+ let foreign_bytes = i128::from(FOREIGN_FILE_BYTES as u64);
+ let idle = UsageSnapshot::take(&http, &data_path).await;
+
+ std::fs::write(&foreign_path, vec![0u8; FOREIGN_FILE_BYTES]).expect("write foreign file");
Review Comment:
nit: this writes 16 MiB without fsync and gives `statvfs` 5 s to move - btrfs only moves it after writeback and a commit, and openzfs after a txg sync. call `syncfs` after the write and after the delete.
##########
core/server/src/http/metrics.rs:
##########
@@ -158,6 +159,55 @@ fn encode_topic_gauge(
Ok(())
}
+/// Exports the disk usage estimate as a gauge, read at encode time like
+/// [`StatsRollupUnderflows`].
+///
+/// The value is the last `statvfs` sample of the volume holding the data
+/// directory plus what Iggy has written since, so it moves with a burst
+/// instead of once a second. No series is exported until the first sample,
+/// since the count alone says nothing about the volume.
+///
+/// It holds the estimate it reads, so a test can export one of its own
+/// instead of the process-wide one other tests charge concurrently.
+#[derive(Debug)]
+struct DiskUsedBytes(&'static DiskUsage);
Review Comment:
simplification: `DiskUsedBytes` and `DiskChargedBytes` wrap the same `DiskUsage` and always register together. merge them into one collector like `TopicUsage`, gating only the gauge on `is_measured()` - the output stays identical.
##########
core/server_common/src/fs_utils.rs:
##########
@@ -257,4 +304,113 @@ mod tests {
});
assert_eq!(std::fs::read(&path).unwrap(), b"messages");
}
+
+ #[cfg(target_os = "linux")]
+ #[test]
+ fn given_a_fresh_file_when_preallocating_should_charge_its_new_blocks_once() {
+ const RESERVATION_BYTES: u64 = 1024 * 1024;
+ const BODY_BYTES: usize = 64 * 1024;
+ let directory = tempfile::tempdir().unwrap();
+ let path = directory.path().join("segment.log");
+ let probe = std::fs::File::create(directory.path().join("probe")).unwrap();
+ if let Err(error) = fallocate(&probe, FallocateFlags::FALLOC_FL_KEEP_SIZE, 0, 1) {
Review Comment:
nit: openzfs returns ok for `FALLOC_FL_KEEP_SIZE` without allocating anything, so this guard lets the test run and fail there. skip unless the probe file's `st_blocks` grow.
##########
core/server_common/src/fs_utils.rs:
##########
@@ -60,7 +64,19 @@ impl DirEntry {
/// Unsupported or failed reservations fall back to buffered allocation.
#[cfg(target_os = "linux")]
pub async fn preallocate_file(file: &fs::File, file_path: &Path, len: u64) {
- let Ok(len) = i64::try_from(len) else {
+ preallocate_charging(&DISK_USAGE, file, file_path, len).await;
+}
+
+/// [`preallocate_file`] charging `usage`, so a test can watch an estimate of
+/// its own instead of the process-wide one other tests charge concurrently.
+#[cfg(target_os = "linux")]
+async fn preallocate_charging(
Review Comment:
simplification: `preallocate_charging` exists only so tests can inject a `DiskUsage`. fold it into `preallocate_file` and have those tests call `reserve` with a local `DiskUsage`, keeping the same counter-delta asserts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6120753317
@andreinnanu right, the sampler interval is capped at 1 s whatever `sysinfo_print_interval` is. i'd drop the freeze i asked for anyway: it needs a server switch whose only job is to turn off the re-base PR2 relies on.
split the counter instead:
- write sites `fetch_add` into a charged total the sampler never touches.
- the sampler stores `base = sample - charged` (wrapping), with `charged` read before `statvfs` as now. `used_bytes()` returns `base + charged`.
- `/metrics` exports the total as a counter, e.g. `disk_charged_bytes` (prometheus-client adds `_total`).
the burst test then compares the counter delta with the `allocated_bytes(data_path)` delta within a stated bound, reading both once the counter stops moving. keep it under the 128 MiB WAL checkpoint, because the counter never sees frees. the sampler keeps running and the test can run beside others. it also replaces the between-samples test, whose 1 MiB tolerance can't see a missing WAL charge.
also run it with `preallocate_segments` on. writes into a reserved segment are charged twice: `preallocate_file` charges the fallocate, then `write_segment_bodies` / `write_frozen_chunked` charge the same bytes again. on 89fb465 an 8 MiB burst into such a topic leaves `disk_used_bytes` 8 MiB above `df` until the next sample. charge only what lands past the reservation, and reset the reserved length on truncate, since ftruncate frees the reservation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub) via commits" <co...@iggy.apache.org>.
andreinnanu commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4477915110
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
I did not consider multiple types of filesystems. Is there an official list that Iggy supports?
I tried to look in the docs, but no luck so far.
I would like to cover all the supported ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6296021874
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4488?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `95.89844%` with `21 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 89.71%. Comparing base ([`c5709f5`](https://app.codecov.io/gh/apache/iggy/commit/c5709f52042477d24c0847e6035bfdcafddc32bc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f3d01a9`](https://app.codecov.io/gh/apache/iggy/commit/f3d01a90b2a736e7dc9eabc6e9974e1f5d34b8b4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/4488?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [core/partitions/src/offset\_storage.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Foffset_storage.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9vZmZzZXRfc3RvcmFnZS5ycw==) | 87.50% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/server\_common/src/fs\_utils.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fserver_common%2Fsrc%2Ffs_utils.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXJfY29tbW9uL3NyYy9mc191dGlscy5ycw==) | 95.06% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/journal/src/partition\_journal.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fjournal%2Fsrc%2Fpartition_journal.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9qb3VybmFsL3NyYy9wYXJ0aXRpb25fam91cm5hbC5ycw==) | 92.30% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/server\_common/src/disk\_usage.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fserver_common%2Fsrc%2Fdisk_usage.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXJfY29tbW9uL3NyYy9kaXNrX3VzYWdlLnJz) | 98.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/server/src/http/metrics.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fhttp%2Fmetrics.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL2h0dHAvbWV0cmljcy5ycw==) | 96.61% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/journal/src/partition\_journal/segments.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fjournal%2Fsrc%2Fpartition_journal%2Fsegments.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9qb3VybmFsL3NyYy9wYXJ0aXRpb25fam91cm5hbC9zZWdtZW50cy5ycw==) | 98.07% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/partitions/src/segment\_recovery.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fsegment_recovery.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9zZWdtZW50X3JlY292ZXJ5LnJz) | 80.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/server/src/sysinfo\_probe.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fsysinfo_probe.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL3N5c2luZm9fcHJvYmUucnM=) | 85.71% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
:x: Your patch check has failed because the patch coverage (23.46%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the [target](https://docs.codecov.com/docs/commit-status#target) coverage.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4488 +/- ##
============================================
+ Coverage 89.70% 89.71% +0.01%
+ Complexity 2146 2145 -1
============================================
Files 1343 1344 +1
Lines 289946 290470 +524
Branches 246223 246703 +480
============================================
+ Hits 260097 260597 +500
- Misses 24138 24166 +28
+ Partials 5711 5707 -4
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.95% <95.89%> (+<0.01%)` | :arrow_up: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.17% <ø> (-0.02%)` | :arrow_down: |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.39% <ø> (+0.04%)` | :arrow_up: |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.71% <ø> (+0.04%)` | :arrow_up: |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.87% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `95.23% <ø> (-0.01%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.88% <ø> (+0.05%)` | :arrow_up: |
| [C++ SDK](https://app.codecov.io/gh/apache/iggy/pull/4488/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.69% <ø> (ø)` | |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/4488?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/journal/src/durable\_storage.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fjournal%2Fsrc%2Fdurable_storage.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9qb3VybmFsL3NyYy9kdXJhYmxlX3N0b3JhZ2UucnM=) | `83.24% <100.00%> (+0.27%)` | :arrow_up: |
| [core/journal/src/file\_storage.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fjournal%2Fsrc%2Ffile_storage.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9qb3VybmFsL3NyYy9maWxlX3N0b3JhZ2UucnM=) | `77.35% <100.00%> (+0.43%)` | :arrow_up: |
| [core/journal/src/prepare\_journal.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fjournal%2Fsrc%2Fprepare_journal.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9qb3VybmFsL3NyYy9wcmVwYXJlX2pvdXJuYWwucnM=) | `93.94% <100.00%> (+<0.01%)` | :arrow_up: |
| [core/partitions/src/iggy\_index\_writer.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Figgy_index_writer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9pZ2d5X2luZGV4X3dyaXRlci5ycw==) | `93.80% <100.00%> (ø)` | |
| [core/partitions/src/messages\_writer.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fmessages_writer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9tZXNzYWdlc193cml0ZXIucnM=) | `92.90% <100.00%> (+0.54%)` | :arrow_up: |
| [core/partitions/src/state\_transfer.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fstate_transfer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9zdGF0ZV90cmFuc2Zlci5ycw==) | `81.79% <100.00%> (+0.01%)` | :arrow_up: |
| [core/journal/src/partition\_journal/segments.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fjournal%2Fsrc%2Fpartition_journal%2Fsegments.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9qb3VybmFsL3NyYy9wYXJ0aXRpb25fam91cm5hbC9zZWdtZW50cy5ycw==) | `87.37% <98.07%> (+0.75%)` | :arrow_up: |
| [core/partitions/src/segment\_recovery.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fsegment_recovery.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9zZWdtZW50X3JlY292ZXJ5LnJz) | `89.65% <80.00%> (+0.01%)` | :arrow_up: |
| [core/server/src/sysinfo\_probe.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fsysinfo_probe.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL3N5c2luZm9fcHJvYmUucnM=) | `92.77% <85.71%> (-0.90%)` | :arrow_down: |
| [core/server/src/http/metrics.rs](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fhttp%2Fmetrics.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL2h0dHAvbWV0cmljcy5ycw==) | `95.44% <96.61%> (+0.25%)` | :arrow_up: |
| ... and [4 more](https://app.codecov.io/gh/apache/iggy/pull/4488?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [33 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4488/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6285890468
@ByteBaker i did a quite a big change to server and all related crates PR #4561. could you please rebase this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4512149138
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6626127355
could you please rebase this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub) via commits" <co...@iggy.apache.org>.
andreinnanu commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4478145996
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
Review Comment:
Yes, this is one of the things I was trying to avoid with the [initial proposal](https://github.com/apache/iggy/issues/4140#issuecomment-6045884268).
I will do the profiling and come back with an answer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4247919210
##########
core/partitions/src/messages_writer.rs:
##########
@@ -196,6 +197,7 @@ async fn write_frozen_chunked<const ALIGN: usize>(
);
IggyError::CannotWriteToFile
})?;
+ DISK_USAGE.charge(chunk_size as u64);
Review Comment:
warning: `preallocate_file` charges the blocks `fallocate` already allocated for a reserved segment (`fs_utils.rs:89`). This line charges the same bytes again as writes fill that segment, so `disk_used_bytes` reports the volume as fuller by the bytes written since the last sample. Charge the reservation or the writes, not both. Also at `core/journal/src/partition_journal/segments.rs:433` and `:451`.
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -423,10 +425,12 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
buffers.push(prepares[index].slice(size_of::<PrepareHeader>()..));
bodies.next();
}
+ let body_bytes = buffers.iter().map(Frozen::len).sum::<usize>() as u64;
Review Comment:
simplification: The new `body_bytes` sum walks `buffers` again, though the push loop above already holds every slice length. Add each length to a counter while pushing, or reuse `end - first.position`, which holds the same total.
##########
core/integration/src/harness/disk.rs:
##########
@@ -114,6 +115,38 @@ pub fn walk(root: &Path, matches: &mut dyn FnMut(&Path) -> bool) -> Option<PathB
None
}
+/// Bytes the filesystem has allocated for `root` and everything under it, the
+/// way `du` counts them: from `st_blocks`, each inode once. File lengths would
+/// count a WAL hard link to a segment body twice and miss the preallocated
+/// tail of a segment.
+pub fn allocated_bytes(root: &Path) -> u64 {
Review Comment:
simplification: `allocated_bytes()` has no caller outside its own two tests, because the disk usage test uses `volume_used_bytes()`. Delete the helper and its tests, or use them in a test that counts data-directory bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6288877984
yes @andreinnanu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4513563139
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -712,7 +757,8 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
if length == 0
Review Comment:
nit: a preallocated tail that `enable_segment_storage` opens with `length > 0` records no reservation here, so `charged_segment_writing_file` charges its body writes again until the next roll. record what its blocks cover, like the retained branch at line 725.
##########
core/server/src/http/metrics.rs:
##########
@@ -158,6 +159,53 @@ fn encode_topic_gauge(
Ok(())
}
+/// Exports the disk usage of the volume holding the data directory, read at
+/// encode time like [`StatsRollupUnderflows`].
+///
+/// `disk_used_bytes` is the estimate: the last `statvfs` sample plus what Iggy
+/// has written since, so it moves with a burst instead of once a second. It is
+/// not exported until the first sample, since the charges alone say nothing
+/// about the volume. Used means total minus free blocks (`f_blocks - f_bfree`),
+/// as `df` reports it. The free space in `/stats` is `f_bavail`, which leaves
+/// out the blocks reserved for root, so the two differ by that reserve.
+///
+/// `disk_allocated_bytes` counts what the write sites charged since the
+/// process started, the part of the estimate that samples never move. Its
+/// increase over an interval follows what Iggy allocated in it, with two
+/// overcounts: a write that failed stays charged, and a preallocation charges
Review Comment:
nit: this says a preallocation charges its whole segment, but `reserve` only charges the growth past blocks the file already owns. say a reservation charges what it adds, and the whole segment only for a new file.
##########
core/journal/src/durable_storage.rs:
##########
@@ -142,9 +142,10 @@ pub trait DurableStorage {
pub trait DurableFile {
/// Best-effort reservation that must not change bytes or logical length.
Review Comment:
nit: the `preallocate` contract does not say that the implementation charges `DISK_USAGE`, or that callers charge only writes past the returned length. spell out both here and on `preallocate_file`.
##########
core/server_common/src/lib.rs:
##########
@@ -42,6 +43,7 @@ pub use consensus_message::{
MutableBacking, RequestBacking, RequestBackingKind, ResponseBacking, ResponseBackingKind,
ResponseFragments,
};
+pub use disk_usage::{Charge, DISK_USAGE, DiskUsage};
Review Comment:
nit: a bare `Charge` at the crate root does not say what it charges, for example `Option<Charge<'static>>` in `offset_storage.rs`. rename it `DiskCharge`, next to `DiskUsage` and `DISK_USAGE`.
##########
core/server/src/http/metrics.rs:
##########
@@ -158,6 +159,53 @@ fn encode_topic_gauge(
Ok(())
}
+/// Exports the disk usage of the volume holding the data directory, read at
+/// encode time like [`StatsRollupUnderflows`].
+///
+/// `disk_used_bytes` is the estimate: the last `statvfs` sample plus what Iggy
+/// has written since, so it moves with a burst instead of once a second. It is
+/// not exported until the first sample, since the charges alone say nothing
+/// about the volume. Used means total minus free blocks (`f_blocks - f_bfree`),
+/// as `df` reports it. The free space in `/stats` is `f_bavail`, which leaves
+/// out the blocks reserved for root, so the two differ by that reserve.
+///
+/// `disk_allocated_bytes` counts what the write sites charged since the
+/// process started, the part of the estimate that samples never move. Its
+/// increase over an interval follows what Iggy allocated in it, with two
+/// overcounts: a write that failed stays charged, and a preallocation charges
+/// its whole segment up front, even when the reservation fails or allocates
+/// nothing and the writes into the segment are then charged again. Unlike
+/// `disk_used_bytes`, it does not move with deletes or with other processes on
+/// the volume.
+///
+/// It holds the estimate it reads, so a test can export one of its own
+/// instead of the process-wide one other tests charge concurrently.
+#[derive(Debug)]
+struct DiskUsageMetrics(&'static DiskUsage);
+
+impl Collector for DiskUsageMetrics {
+ fn encode(&self, mut encoder: DescriptorEncoder) -> Result<(), std::fmt::Error> {
+ if let Some(used_bytes) = self.0.used_bytes() {
+ let gauge = ConstGauge::new(gauge_value(used_bytes));
+ let metric_encoder = encoder.encode_descriptor(
+ "disk_used_bytes",
Review Comment:
nit: `disk_used_bytes` reads like iggy's own share, but it counts every file on the data volume. name it for the volume, like the `volume_used_bytes` test helper, before dashboards depend on it.
##########
core/server/src/http/metrics.rs:
##########
@@ -244,6 +292,7 @@ impl HttpMetrics {
registry.register_collector(Box::new(StatsRollupUnderflows));
let topic_usage = TopicUsage::default();
registry.register_collector(Box::new(topic_usage.clone()));
+ registry.register_collector(Box::new(DiskUsageMetrics(&DISK_USAGE)));
Review Comment:
nit: this exports the disk series to every authenticated user, while `/stats` shows disk space only after `Permissioner::get_stats`. gate it like the per-topic series, unless host volume usage counts as a safe aggregate here.
##########
core/server_common/src/fs_utils.rs:
##########
@@ -257,4 +328,133 @@ mod tests {
});
assert_eq!(std::fs::read(&path).unwrap(), b"messages");
}
+
+ #[cfg(target_os = "linux")]
+ #[test]
+ fn given_a_fresh_file_when_preallocating_should_charge_its_new_blocks_once() {
+ const RESERVATION_BYTES: u64 = 1024 * 1024;
+ const BODY_BYTES: usize = 64 * 1024;
+ let directory = tempfile::tempdir().unwrap();
+ let path = directory.path().join("segment.log");
+ let probe = std::fs::File::create(directory.path().join("probe")).unwrap();
+ let probe_len = i64::try_from(BODY_BYTES).unwrap();
+ // OpenZFS reports success without allocating anything and still counts
+ // a block per file for its dnode, so only blocks that cover the probe count.
+ if fallocate(&probe, FallocateFlags::FALLOC_FL_KEEP_SIZE, 0, probe_len).is_err()
Review Comment:
simplification: this probe is the same fallocate check as `supports_preallocation` in the journal and `reservations_allocate` in the test harness. export one probe from `server_common::fs_utils` and call it from all three.
##########
core/journal/src/partition_journal.rs:
##########
@@ -163,6 +163,9 @@ pub struct PartitionPrepareJournal<S: DurableStorage = DiskStorage> {
cleanup_directory_dirty: bool,
recovered_prepares: Vec<Message<PrepareHeader>>,
segment_files: BTreeMap<(u64, u64), S::File>,
+ /// How much of each open segment file its reservation covers. A file
+ /// without an entry has none, so all its writes take new space.
+ segment_reservations: BTreeMap<(u64, u64), u64>,
Review Comment:
simplification: `segment_reservations` has the same keys and lifetime as `segment_files`, so every insert and `retain` has to be paired by hand. keep the reserved end next to the handle in `segment_files`, like `MessagesWriter` keeps `reserved_end`.
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+//!
+//! Re-basing assumes `statvfs` already counts every write that finished
+//! before the sample. ext4 and XFS do: they take a write's blocks out of the
+//! free count as soon as the write reserves them. btrfs and OpenZFS do not:
+//! they update the free count only when they commit, every 30 s on btrfs
+//! (later for writes still waiting in the page cache) and every 5 s on OpenZFS
+//! by default. On those filesystems the estimate can read low after a sample,
+//! by the writes that finished since the last commit, until the next commit.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It has no value until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since. `None` until a sample sets the base, since the charges
+ /// alone say nothing about the volume.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> Option<u64> {
+ if !self.measured.load(Ordering::Acquire) {
+ return None;
+ }
+ // Pairs with the release in `rebase`, which follows its acquire of
+ // `settled`, which follows each settled charge's `fetch_add`. So the
+ // `charged` load below sees every charge the base subtracted, and the
+ // estimate never reads below the sample. The base must be loaded first.
+ let base = self.base.load(Ordering::Acquire);
+ Some(base.wrapping_add(self.charged.load(Ordering::Relaxed)))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ /// `/metrics` exports it as `disk_allocated_bytes`.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ ///
+ /// On btrfs and OpenZFS the volume shows a finished write only after its
+ /// commit, so the estimate reads low until then (see the module docs).
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
+ let (sample, used_bytes) = measure()?;
+ self.base
+ .store(used_bytes.wrapping_sub(settled), Ordering::Release);
+ self.measured.store(true, Ordering::Release);
+ Some(sample)
+ }
+}
+
+impl Default for DiskUsage {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Bytes charged before they are written. Dropping it, once the write finished
+/// or failed, lets the next sample take them over: a finished write is on the
+/// volume, and a failed one allocated only what the volume shows.
+#[derive(Debug)]
+#[must_use = "a charge settles when dropped, so hold it until the write finishes"]
+pub struct Charge<'a> {
+ usage: &'a DiskUsage,
+ bytes: u64,
+}
+
+impl Drop for Charge<'_> {
+ #[inline]
+ fn drop(&mut self) {
+ if self.bytes > 0 {
+ self.usage.settled.fetch_add(self.bytes, Ordering::Release);
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use std::sync::Barrier;
+ use std::thread;
+
+ const MIB: u64 = 1024 * 1024;
+
+ #[test]
+ fn given_no_sample_when_charging_should_have_no_estimate_yet() {
+ let usage = DiskUsage::new();
+
+ drop(usage.charge(3 * MIB));
+
+ assert_eq!(usage.used_bytes(), None);
+ assert_eq!(usage.charged_bytes(), 3 * MIB);
+ }
+
+ #[test]
+ fn given_charges_when_rebasing_should_replace_them_with_the_sample() {
+ let usage = DiskUsage::new();
+ drop(usage.charge(5 * MIB));
+
+ let sample = usage.rebase(|| Some(("sample", 100 * MIB)));
+
+ assert_eq!(sample, Some("sample"));
+ assert_eq!(usage.used_bytes(), Some(100 * MIB));
+ }
+
+ #[test]
+ fn given_a_sample_below_the_estimate_when_rebasing_should_lower_it() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ drop(usage.charge(40 * MIB));
+
+ usage.rebase(|| Some(((), 90 * MIB)));
+
+ assert_eq!(
+ usage.used_bytes(),
+ Some(90 * MIB),
+ "space freed since the last sample must lower the estimate"
+ );
+ }
+
+ #[test]
+ fn given_a_charge_from_another_thread_after_the_measurement_when_rebasing_should_keep_it() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ let measured = Barrier::new(2);
+ let charged = Barrier::new(2);
+
+ thread::scope(|scope| {
+ scope.spawn(|| {
+ measured.wait();
+ drop(usage.charge(MIB));
+ charged.wait();
+ });
+ usage.rebase(|| {
+ // The volume is measured before the writer charges, so the
+ // sample cannot hold its bytes; the base is stored after.
+ let sample = 100 * MIB;
+ measured.wait();
+ charged.wait();
+ Some(((), sample))
+ });
+ });
+
+ assert_eq!(
+ usage.used_bytes(),
+ Some(101 * MIB),
+ "storing the base must not drop a charge that landed after the measurement"
+ );
+ assert_eq!(usage.charged_bytes(), MIB);
+ }
+
+ #[test]
+ fn given_a_charge_during_the_sample_when_rebasing_should_count_it_until_the_next_sample() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+
+ usage.rebase(|| {
+ drop(usage.charge(MIB));
+ Some(((), 100 * MIB))
+ });
+
+ assert_eq!(
+ usage.used_bytes(),
+ Some(101 * MIB),
+ "a charge the sample may have missed must still be counted"
+ );
+ }
+
+ #[test]
+ fn given_a_write_in_flight_when_rebasing_should_keep_its_charge_until_it_settles() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ let write = usage.charge(MIB);
+
+ // The write has not reached the volume, so the sample does not hold it.
+ usage.rebase(|| Some(((), 100 * MIB)));
+
+ assert_eq!(
+ usage.used_bytes(),
+ Some(101 * MIB),
+ "a sample taken while a write is in flight must not drop its charge"
+ );
+
+ drop(write);
+ usage.rebase(|| Some(((), 101 * MIB)));
+
+ assert_eq!(
+ usage.used_bytes(),
+ Some(101 * MIB),
+ "the first sample that holds a settled write must replace its charge"
+ );
+ }
+
+ #[test]
+ fn given_a_write_in_flight_on_another_thread_when_rebasing_should_keep_its_charge() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ let charged = Barrier::new(2);
+ let sampled = Barrier::new(2);
+
+ thread::scope(|scope| {
+ scope.spawn(|| {
+ let write = usage.charge(MIB);
+ charged.wait();
+ // The bytes reach the volume only after the sample.
+ sampled.wait();
+ drop(write);
+ });
+ charged.wait();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ sampled.wait();
+ });
+
+ assert_eq!(
+ usage.used_bytes(),
+ Some(101 * MIB),
+ "a sample must not drop the charge of a write that was still in flight"
+ );
+ }
+
+ #[test]
+ fn given_a_failed_write_when_its_charge_settles_should_let_the_next_sample_remove_it() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ let write = usage.charge(MIB);
+
+ // The write failed before allocating anything.
+ drop(write);
+ usage.rebase(|| Some(((), 100 * MIB)));
+
+ assert_eq!(usage.used_bytes(), Some(100 * MIB));
+ assert_eq!(usage.charged_bytes(), MIB);
+ }
+
+ #[test]
+ fn given_a_failed_sample_when_rebasing_should_keep_counting_from_the_previous_base() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ drop(usage.charge(MIB));
+
+ let sample: Option<()> = usage.rebase(|| None);
+
+ assert!(sample.is_none());
+ assert_eq!(usage.used_bytes(), Some(101 * MIB));
+ }
+
+ #[test]
+ fn given_samples_between_charges_when_reading_the_charged_total_should_count_only_charges() {
+ let usage = DiskUsage::new();
+ drop(usage.charge(5 * MIB));
+ usage.rebase(|| Some(((), 100 * MIB)));
+ drop(usage.charge(2 * MIB));
+ usage.rebase(|| Some(((), 50 * MIB)));
+
+ assert_eq!(usage.charged_bytes(), 7 * MIB);
+ assert_eq!(usage.used_bytes(), Some(50 * MIB));
+ }
+
+ #[test]
+ fn given_charges_past_u64_max_when_reading_the_charged_total_should_wrap_instead_of_panicking()
Review Comment:
simplification: this test and the ones at lines 358 and 374 push `charged` past `u64::MAX`, which takes 16 EiB of writes, and `fetch_add` wraps by definition. drop them and keep the line 387 test for the reachable `base` wrap.
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -436,19 +442,37 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
reference: SegmentReference,
prepare: &Frozen<4096>,
) -> io::Result<()> {
- self.segment_writing_file(reference)
- .await?
- .write_frozen(
- reference.position,
- prepare.slice(size_of::<PrepareHeader>()..),
- )
- .await
+ let end = reference.position + (prepare.len() - size_of::<PrepareHeader>()) as u64;
+ let (_charge, file) = self.charged_segment_writing_file(reference, end).await?;
+ file.write_frozen(
+ reference.position,
+ prepare.slice(size_of::<PrepareHeader>()..),
+ )
+ .await?;
+ Ok(())
+ }
+
+ /// How many bytes of a body write at `position..end` of `segment` take new
+ /// disk space. Writes below the end of its reservation take none. The part
+ /// past it does: a segment can grow past `max_size` because the last batch
+ /// may cross it, and without a reservation that is the whole write.
+ fn newly_allocated(&self, segment: (u64, u64), position: u64, end: u64) -> u64 {
Review Comment:
simplification: `newly_allocated` repeats the `end.saturating_sub(position.max(reserved_end))` rule from `write_frozen_chunked` in `messages_writer.rs`, and three tests open a WAL just to check it. move it into one `const fn` in `server_common::disk_usage` with one plain unit test.
##########
core/server/src/http/metrics.rs:
##########
@@ -158,6 +159,53 @@ fn encode_topic_gauge(
Ok(())
}
+/// Exports the disk usage of the volume holding the data directory, read at
+/// encode time like [`StatsRollupUnderflows`].
+///
+/// `disk_used_bytes` is the estimate: the last `statvfs` sample plus what Iggy
+/// has written since, so it moves with a burst instead of once a second. It is
+/// not exported until the first sample, since the charges alone say nothing
+/// about the volume. Used means total minus free blocks (`f_blocks - f_bfree`),
+/// as `df` reports it. The free space in `/stats` is `f_bavail`, which leaves
+/// out the blocks reserved for root, so the two differ by that reserve.
+///
+/// `disk_allocated_bytes` counts what the write sites charged since the
+/// process started, the part of the estimate that samples never move. Its
+/// increase over an interval follows what Iggy allocated in it, with two
+/// overcounts: a write that failed stays charged, and a preallocation charges
+/// its whole segment up front, even when the reservation fails or allocates
+/// nothing and the writes into the segment are then charged again. Unlike
+/// `disk_used_bytes`, it does not move with deletes or with other processes on
+/// the volume.
+///
+/// It holds the estimate it reads, so a test can export one of its own
+/// instead of the process-wide one other tests charge concurrently.
+#[derive(Debug)]
+struct DiskUsageMetrics(&'static DiskUsage);
+
+impl Collector for DiskUsageMetrics {
+ fn encode(&self, mut encoder: DescriptorEncoder) -> Result<(), std::fmt::Error> {
+ if let Some(used_bytes) = self.0.used_bytes() {
+ let gauge = ConstGauge::new(gauge_value(used_bytes));
Review Comment:
simplification: `used_bytes` goes through the `gauge_value` clamp, whose doc says only `messages` needs it. use `ConstGauge::new(used_bytes)`, since `ConstGauge<u64>` already encodes, as in `encode_topic_gauge`.
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -592,8 +621,11 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
file.truncate(cursor.position.length).await?;
self.segment_files_dirty = true;
if self.preallocate_segments {
- file.preallocate(&cursor.path(&self.directory), segments.max_size)
+ let reserved = file
Review Comment:
simplification: on boot a truncated tail gets reserved twice - once inside `open_segment_file` at line 726, then again here. pass `None` to `open_segment_file` in `recover_segment_files` and reserve once after the optional truncation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "justinmclean (via GitHub)" <gi...@apache.org>.
justinmclean commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6119847123
Thanks @andreinnanu for this. @hubcio, can you answer the open question on freezing the sampler for the burst test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub)" <gi...@apache.org>.
andreinnanu commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6324625763
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4478165566
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
just stick to popular ones - ext4/btrfx/xfs/zfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6112183837
<!-- iggy-pr-triage-welcome -->
Thanks for the PR. It is labeled `S-waiting-on-review` and queued for review.
Slash commands (own line, regular comment) move it around the queue:
- `/ready` - back to `S-waiting-on-review` after addressing feedback
- `/author` - flip to `S-waiting-on-author` while you finish changes
- `/request-review @user-or-team` - request a reviewer
- `/pin` - exempt the PR from the stale bot, `/unpin` to undo
See [CONTRIBUTING.md](https://github.com/apache/iggy/blob/master/CONTRIBUTING.md#pr-triage-commands) for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4249012271
##########
core/journal/src/durable_storage.rs:
##########
@@ -64,6 +65,10 @@ pub struct StorageEntry {
pub trait DurableStorage {
type File: DurableFile;
+ /// Count `bytes` just allocated on disk towards the process-wide disk
+ /// usage estimate. Backends that do not write to the data volume ignore it.
+ fn charge_allocation(&self, _bytes: u64) {}
Review Comment:
warning: `persist_offset_retained` and `persist_offset_with_storage` create offset files without charging this counter, so usage is understated between samples. charge new offset allocations without counting existing blocks again.
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -423,10 +425,12 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
buffers.push(prepares[index].slice(size_of::<PrepareHeader>()..));
bodies.next();
}
+ let body_bytes = buffers.iter().map(Frozen::len).sum::<usize>() as u64;
self.segment_writing_file(first)
.await?
.write_frozen_vectored(first.position, buffers)
.await?;
+ self.storage.charge_allocation(body_bytes);
Review Comment:
warning: writes into preallocated space charge blocks that the reservation already counted, so `disk_used_bytes` grows without equivalent allocation. charge only new allocation here and in `MessagesWriter`.
##########
core/server_common/src/fs_utils.rs:
##########
@@ -257,4 +278,36 @@ mod tests {
});
assert_eq!(std::fs::read(&path).unwrap(), b"messages");
}
+
+ #[cfg(target_os = "linux")]
+ #[test]
+ fn given_a_reserved_file_when_reserving_again_should_report_nothing_newly_allocated() {
Review Comment:
warning: this test calls `reserve` directly and never checks `DISK_USAGE`, so it still passes if the production charge disappears. assert counter deltas for fresh reservations, repeated reservations, and writes into reserved space.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -4537,11 +4537,13 @@ pub(crate) async fn reclaim_retry_checkpoint_files(
}
async fn write_staging_file(path: &Path, payload: Vec<u8>) -> std::io::Result<()> {
+ let len = payload.len() as u64;
let mut file = compio::fs::File::create(path)
.await
.note_descriptor_exhaustion(|| format!("creating {}", path.display()))?;
let (result, _) = file.write_all_at(payload, 0).await.into();
result?;
+ DISK_USAGE.charge(len);
Review Comment:
warning: a partial write can allocate bytes before returning an error, but this path skips the charge and leaves those bytes uncounted. reconcile allocation after failed writes here and in the journal, message, and index writers.
##########
core/server/src/http/metrics.rs:
##########
@@ -270,6 +298,23 @@ mod tests {
);
}
+ /// Shares the process-wide estimate with every other test in this binary,
+ /// so this asserts the series and its type once a sample exists.
+ #[test]
+ fn disk_used_bytes_lands_in_the_exposition_once_measured() {
Review Comment:
warning: this test checks the metric name and type but never its value, so a collector that emits a constant passes. assert the sampled value and its change after a charge.
##########
core/integration/tests/server/disk_usage.rs:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The disk usage estimate behind `disk_used_bytes` against `statvfs` of the
+//! data volume. Both read the whole volume, so these tests must run alone: a
+//! `#[serial]` excludes the harness tests under `cargo test`, and
+//! `.config/nextest.toml` gives them every thread under nextest.
+
+use std::time::{Duration, Instant};
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::harness::disk::volume_used_bytes;
+use integration::harness::seeds::{self, names};
+use serial_test::serial;
+use tokio::time::sleep;
+
+use super::http_client::HttpClient;
+
+const KIB: u64 = 1024;
+const MIB: u64 = 1024 * KIB;
+/// Per round the server writes about 2 MiB of messages, and more than that
+/// once the WAL and the segment both hold them. An estimate that missed the
+/// write sites would be off by at least that much until the next sample.
+const MESSAGES_PER_ROUND: usize = 16;
+const MESSAGE_BYTES: usize = 128 * KIB as usize;
+const ROUNDS: usize = 8;
+/// Block rounding, file metadata and the server's own log lines, none of
+/// which the write sites charge.
+const TOLERANCE: u64 = MIB;
+const FOREIGN_FILE_BYTES: usize = 16 * MIB as usize;
+/// Comfortably more than one sampling interval (1 s).
+const SAMPLE_DEADLINE: Duration = Duration::from_secs(5);
+const POLL_INTERVAL: Duration = Duration::from_millis(100);
+
+#[tokio::test]
+#[serial]
+async fn given_iggy_writes_between_samples_when_scraping_should_track_the_volume_within_tolerance()
+{
+ let harness = start_server().await;
+ let client = harness.tcp_root_client().await.expect("tcp root client");
+ let http = HttpClient::login_root(&harness).await;
+ let data_path = harness.server().data_path();
+ let stream = Identifier::named(names::STREAM).expect("stream identifier");
+ let topic = Identifier::named(names::TOPIC).expect("topic identifier");
+
+ let mut rounds = Vec::with_capacity(ROUNDS);
+ for round in 0..ROUNDS {
+ let mut messages: Vec<IggyMessage> = (0..MESSAGES_PER_ROUND)
+ .map(|_| {
+ IggyMessage::builder()
+ .payload(vec![round as u8; MESSAGE_BYTES].into())
+ .build()
+ .expect("build message")
+ })
+ .collect();
+ client
+ .send_messages(
+ &stream,
+ &topic,
+ &Partitioning::partition_id(0),
+ &mut messages,
+ )
+ .await
+ .expect("send messages");
+
+ // Bracketing the scrape absorbs writes the server finishes while it
+ // runs, such as a segment flush that follows the WAL write.
+ let before = volume_used_bytes(&data_path);
+ let estimate = scrape_estimate(&http).await;
+ let after = volume_used_bytes(&data_path);
+ rounds.push((before, estimate, after));
+ }
+
+ let outside: Vec<_> = rounds
+ .iter()
+ .enumerate()
+ .filter(|(_, (before, estimate, after))| {
+ *estimate + TOLERANCE < *before || *estimate > *after + TOLERANCE
+ })
+ .collect();
+ assert!(
+ outside.is_empty(),
+ "the estimate must stay within {TOLERANCE} bytes of statvfs, rounds outside \
+ (round, (statvfs before, estimate, statvfs after)): {outside:?}"
+ );
+}
+
+#[tokio::test]
+#[serial]
+async fn given_a_file_written_outside_iggy_when_the_next_sample_runs_should_count_it_until_deleted()
+{
+ let harness = start_server().await;
+ let http = HttpClient::login_root(&harness).await;
+ let data_path = harness.server().data_path();
+ // Beside the data directory, so on the same volume but never charged.
+ let foreign_path = data_path.with_extension("foreign");
+ let idle = scrape_estimate(&http).await;
+
+ std::fs::write(&foreign_path, vec![0u8; FOREIGN_FILE_BYTES]).expect("write foreign file");
+ let foreign_bytes = FOREIGN_FILE_BYTES as u64;
+ let with_file = wait_for_estimate(&http, |estimate| {
+ estimate + TOLERANCE >= idle + foreign_bytes
Review Comment:
warning: these one-sided bounds accept exaggerated counter changes and never bound the error against `statvfs`. compare counter and filesystem deltas for both file creation and removal.
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! count, so anything the write sites cannot see (hard links, temporary files
+//! replaced by rename, logs, other processes, deleted segments) is wrong for
+//! at most one sampling interval.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to it, so it gets a cache line of its own.
+ used_bytes: CachePadded<AtomicU64>,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ used_bytes: CachePadded::new(AtomicU64::new(0)),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy has just written to, or reserved on, the volume.
+ #[inline]
+ pub fn charge(&self, bytes: u64) {
+ self.used_bytes.fetch_add(bytes, Ordering::Relaxed);
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.used_bytes.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The correction is added instead of stored, so a charge that lands after
+ /// `measure` returns is kept. A charge that lands while `measure` runs is
+ /// counted twice until the next sample, which errs towards a fuller disk.
+ /// When `measure` returns `None` the estimate keeps counting from the
+ /// previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ let counted = self.used_bytes.load(Ordering::Relaxed);
+ let (sample, used_bytes) = measure()?;
+ self.used_bytes
+ .fetch_add(used_bytes.wrapping_sub(counted), Ordering::Relaxed);
+ self.measured.store(true, Ordering::Release);
+ Some(sample)
+ }
+}
+
+impl Default for DiskUsage {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const MIB: u64 = 1024 * 1024;
+
+ #[test]
+ fn given_no_sample_when_charging_should_count_from_zero() {
+ let usage = DiskUsage::new();
+
+ usage.charge(3 * MIB);
+
+ assert_eq!(usage.used_bytes(), 3 * MIB);
+ assert!(!usage.is_measured());
+ }
+
+ #[test]
+ fn given_charges_when_rebasing_should_replace_them_with_the_sample() {
+ let usage = DiskUsage::new();
+ usage.charge(5 * MIB);
+
+ let sample = usage.rebase(|| Some(("sample", 100 * MIB)));
+
+ assert_eq!(sample, Some("sample"));
+ assert_eq!(usage.used_bytes(), 100 * MIB);
+ assert!(usage.is_measured());
+ }
+
+ #[test]
+ fn given_a_sample_below_the_estimate_when_rebasing_should_lower_it() {
+ let usage = DiskUsage::new();
+ usage.rebase(|| Some(((), 100 * MIB)));
+ usage.charge(40 * MIB);
+
+ usage.rebase(|| Some(((), 90 * MIB)));
+
+ assert_eq!(
+ usage.used_bytes(),
+ 90 * MIB,
+ "space freed since the last sample must lower the estimate"
+ );
+ }
+
+ #[test]
+ fn given_a_charge_after_the_sample_when_rebasing_should_keep_it() {
Review Comment:
warning: this test charges only after `rebase` returns, so it misses updates between measurement and correction. use barriers to place a concurrent charge in that interval.
##########
core/journal/src/partition_journal/segments.rs:
##########
@@ -423,10 +425,12 @@ impl<S: DurableStorage> PartitionPrepareJournal<S> {
buffers.push(prepares[index].slice(size_of::<PrepareHeader>()..));
bodies.next();
}
+ let body_bytes = buffers.iter().map(Frozen::len).sum::<usize>() as u64;
Review Comment:
simplification: this loop sums buffer lengths even though `end` already contains their combined extent. use `end - first.position` after the successful write.
##########
core/integration/tests/server/disk_usage.rs:
##########
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The disk usage estimate behind `disk_used_bytes` against `statvfs` of the
+//! data volume. Both read the whole volume, so these tests must run alone: a
+//! `#[serial]` excludes the harness tests under `cargo test`, and
+//! `.config/nextest.toml` gives them every thread under nextest.
+
+use std::time::{Duration, Instant};
+
+use iggy::prelude::*;
+use integration::harness::TestHarness;
+use integration::harness::disk::volume_used_bytes;
+use integration::harness::seeds::{self, names};
+use serial_test::serial;
+use tokio::time::sleep;
+
+use super::http_client::HttpClient;
+
+const KIB: u64 = 1024;
+const MIB: u64 = 1024 * KIB;
+/// Per round the server writes about 2 MiB of messages, and more than that
+/// once the WAL and the segment both hold them. An estimate that missed the
+/// write sites would be off by at least that much until the next sample.
+const MESSAGES_PER_ROUND: usize = 16;
+const MESSAGE_BYTES: usize = 128 * KIB as usize;
+const ROUNDS: usize = 8;
+/// Block rounding, file metadata and the server's own log lines, none of
+/// which the write sites charge.
+const TOLERANCE: u64 = MIB;
+const FOREIGN_FILE_BYTES: usize = 16 * MIB as usize;
+/// Comfortably more than one sampling interval (1 s).
+const SAMPLE_DEADLINE: Duration = Duration::from_secs(5);
+const POLL_INTERVAL: Duration = Duration::from_millis(100);
+
+#[tokio::test]
+#[serial]
+async fn given_iggy_writes_between_samples_when_scraping_should_track_the_volume_within_tolerance()
+{
+ let harness = start_server().await;
+ let client = harness.tcp_root_client().await.expect("tcp root client");
+ let http = HttpClient::login_root(&harness).await;
+ let data_path = harness.server().data_path();
+ let stream = Identifier::named(names::STREAM).expect("stream identifier");
+ let topic = Identifier::named(names::TOPIC).expect("topic identifier");
+
+ let mut rounds = Vec::with_capacity(ROUNDS);
+ for round in 0..ROUNDS {
+ let mut messages: Vec<IggyMessage> = (0..MESSAGES_PER_ROUND)
+ .map(|_| {
+ IggyMessage::builder()
+ .payload(vec![round as u8; MESSAGE_BYTES].into())
+ .build()
+ .expect("build message")
+ })
+ .collect();
+ client
+ .send_messages(
+ &stream,
+ &topic,
+ &Partitioning::partition_id(0),
+ &mut messages,
+ )
+ .await
+ .expect("send messages");
+
+ // Bracketing the scrape absorbs writes the server finishes while it
+ // runs, such as a segment flush that follows the WAL write.
+ let before = volume_used_bytes(&data_path);
+ let estimate = scrape_estimate(&http).await;
+ let after = volume_used_bytes(&data_path);
+ rounds.push((before, estimate, after));
+ }
+
+ let outside: Vec<_> = rounds
+ .iter()
+ .enumerate()
+ .filter(|(_, (before, estimate, after))| {
+ *estimate + TOLERANCE < *before || *estimate > *after + TOLERANCE
Review Comment:
warning: the sampler can refresh before each assertion, so this test can pass with missing write charges. freeze the sampler and compare counter and filesystem deltas across the same interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio merged PR #4488:
URL: https://github.com/apache/iggy/pull/4488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6123932867
/skill team-review-slim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4478165566
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
just stick to popular ones - ext4/btrfx/xfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4478174518
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
Review Comment:
sure - thats why i treat it as a nit. i dont think there will be visible impact until we hit 10M+++ calls/s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub)" <gi...@apache.org>.
andreinnanu commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6300546711
> yes @andreinnanu
rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub) via commits" <co...@iggy.apache.org>.
andreinnanu commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4506518696
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
After working on it for a while, I chose to document the assumption and fix it in a future issue.
Should I write it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub) via commits" <co...@iggy.apache.org>.
andreinnanu commented on code in PR #4488:
URL: https://github.com/apache/iggy/pull/4488#discussion_r4513356051
##########
core/server_common/src/disk_usage.rs:
##########
@@ -0,0 +1,386 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Estimated usage of the volume holding the data directory.
+//!
+//! A periodic `statvfs` sample sets the base and the sites that allocate disk
+//! space add what they write, so the estimate follows a burst between two
+//! samples without a syscall on the write path. Each sample re-bases the
+//! estimate, so anything the write sites cannot see (hard links, temporary
+//! files replaced by rename, logs, other processes, deleted segments) is wrong
+//! for at most one sampling interval.
+//!
+//! The write sites add to a total of their own that samples never touch, and
+//! a sample only moves the base under it. The total then says what Iggy
+//! charged, apart from what the volume did, which is how a test checks that
+//! every write site charges while the sampler keeps running.
+//!
+//! A write site charges before it writes, so a sample taken while the write is
+//! in flight does not hold its bytes yet. Each charge therefore returns a
+//! [`Charge`], and a sample only takes over the charges whose [`Charge`] was
+//! dropped, once their writes finished or failed.
+
+use crossbeam::utils::CachePadded;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+
+/// The process-wide estimate. It counts from zero until the first sample.
+pub static DISK_USAGE: DiskUsage = DiskUsage::new();
+
+#[derive(Debug)]
+pub struct DiskUsage {
+ // Every shard adds to both, so each gets a cache line of its own.
+ charged: CachePadded<AtomicU64>,
+ /// The part of `charged` whose writes finished or failed.
+ settled: CachePadded<AtomicU64>,
+ /// The last sample minus what was settled before it. Charges outgrow the
+ /// volume once files are deleted, so it wraps below zero and the
+ /// estimate adds the charges back with wrapping arithmetic.
+ base: AtomicU64,
+ measured: AtomicBool,
+}
+
+impl DiskUsage {
+ #[must_use]
+ pub const fn new() -> Self {
+ Self {
+ charged: CachePadded::new(AtomicU64::new(0)),
+ settled: CachePadded::new(AtomicU64::new(0)),
+ base: AtomicU64::new(0),
+ measured: AtomicBool::new(false),
+ }
+ }
+
+ /// Adds `bytes` that Iggy is about to write to, or has reserved on, the
+ /// volume. Hold the returned [`Charge`] until the write finishes or fails.
+ #[inline]
+ pub fn charge(&self, bytes: u64) -> Charge<'_> {
+ if bytes > 0 {
+ self.charged.fetch_add(bytes, Ordering::Relaxed);
+ }
+ Charge { usage: self, bytes }
+ }
+
+ /// Estimated bytes used on the volume: the last sample plus the bytes
+ /// charged since.
+ #[must_use]
+ #[inline]
+ pub fn used_bytes(&self) -> u64 {
+ self.base
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.charged.load(Ordering::Relaxed))
+ }
+
+ /// Every byte charged since the process started. Samples do not move it.
+ #[must_use]
+ #[inline]
+ pub fn charged_bytes(&self) -> u64 {
+ self.charged.load(Ordering::Relaxed)
+ }
+
+ /// Whether a sample has set the base. Until then [`Self::used_bytes`]
+ /// holds only the bytes charged since the process started.
+ #[must_use]
+ pub fn is_measured(&self) -> bool {
+ self.measured.load(Ordering::Acquire)
+ }
+
+ /// Runs `measure` and re-bases the estimate on the used bytes it returns,
+ /// passing its sample through. Call it from one thread only.
+ ///
+ /// The base subtracts only the charges settled before `measure` runs,
+ /// because their writes are on the volume when it is measured. A charge
+ /// whose write is still in flight stays in the estimate, and so does a
+ /// charge that lands after `measure` returns. A write that settles while
+ /// `measure` runs is counted twice until the next sample, which errs
+ /// towards a fuller disk. When `measure` returns `None` the estimate keeps
+ /// counting from the previous base.
+ pub fn rebase<T>(&self, measure: impl FnOnce() -> Option<(T, u64)>) -> Option<T> {
+ // Pairs with the release in `Charge::drop`: every write settled here
+ // finished before the volume is measured.
+ let settled = self.settled.load(Ordering::Acquire);
Review Comment:
Should be fixed in https://github.com/apache/iggy/issues/4807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub)" <gi...@apache.org>.
andreinnanu commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6276044877
Tested by manually freezing the sampler interval (adding a big extra sleep in the binary for testing purposes):
Test scripts:
- measuring script, with a 0.2s interval, checking the /metrics endpoint and `du` for data dir size
- 5 producers, producing data over 5 different topics
```
sending 32 x 65536 B per request to disk-lab/{burst-0, burst-1, burst-2, burst-3, burst-4}, partition 0, rate 64.00 MiB/s
batches 976 sent 1952.00 MiB rate 16.27 MiB/s
done: 976 requests, 1952.00 MiB in 120.01 s (16.27 MiB/s)
```
- 10 consumers, consuming the produced data
```
10 consumers polling disk-lab/burst-0..4, 1 partition(s) each, 16 messages per poll
polls 8246 messages 131936 read 8246.00 MiB 68.52 MiB/s
done: 8246 polls, 131936 messages, 8246.00 MiB in 120.34 s
```
Results:
- Preallocation off
```
largest overcount (estimate grew more than the data directory): 34.18 MiB
largest undercount (estimate grew less than the data directory): 0.05 MiB
```
- Preallocation on
```
largest overcount (estimate grew more than the data directory): 209.39 MiB
largest undercount (estimate grew less than the data directory): 0.04 MiB
```
Because of different frees that are not taken into account, the overcount has the tendency to grow between samples, when it gets reseted. Is this something we should consider at this point?
Also, the large overcounts are usually spikes created by the measuring script, the diff looks like this over time:
```
23:28:42.155 983.45 966.20 17.25
23:28:42.475 983.50 966.22 17.28
23:28:42.772 983.59 966.32 17.27
23:28:43.033 983.64 966.38 17.26
23:28:43.369 1175.90 966.52 209.39
23:28:43.768 1175.99 1158.70 17.30
23:28:44.053 1176.04 1158.79 17.26
23:28:44.397 1176.19 1158.93 17.26
23:28:44.659 1176.24 1158.99 17.26
23:28:44.947 1176.35 1159.09 17.26
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): disk usage accounting [iggy]
Posted by "andreinnanu (via GitHub)" <gi...@apache.org>.
andreinnanu commented on PR #4488:
URL: https://github.com/apache/iggy/pull/4488#issuecomment-6288865068
> @andreinnanu i did a quite a big change to server and all related crates in PR #4561. could you please rebase this PR?
Yes, i can take care of that. Rebasing with current master should be good enough, right @hubcio?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org