You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/29 23:42:39 UTC

(datafusion) branch main updated: feat: log drift between MemoryPool reservations and allocated bytes in SLT (#25651)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git


The following commit(s) were added to refs/heads/main by this push:
     new 1d9be2e107 feat: log drift between MemoryPool reservations and allocated bytes in SLT (#25651)
1d9be2e107 is described below

commit 1d9be2e10794994fc1495c35363658dc7db0adf7
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Sep 29 23:31:56 2026 +0000

    feat: log drift between MemoryPool reservations and allocated bytes in SLT (#25651)
    
    ## Which issue does this PR close?
    
    - Part of #25650.
    
    ## Rationale for this change
    
    `MemoryPool` accounting only covers memory that operators explicitly
    reserve, so a process can be OOM killed while the pool reports plenty of
    headroom. As a first step, this makes the gap visible: it logs the
    difference between what pools have reserved and what's actually
    allocated, and it's on by default in SLT. It only logs and never fails a
    test, unlike #22626 which was reverted in #22860.
    
    ## What changes are included in this PR?
    
    - `datafusion-execution`: `MemoryDriftTracker` and `DriftLoggingPool`.
    The pool wraps any `MemoryPool` and reports reservation changes to a
    tracker, which can be shared by many pools. The tracker compares the
    reserved total against allocated bytes from a caller-supplied `Fn() ->
    usize`, since DataFusion doesn't choose the global allocator. It logs at
    `info` each time positive drift rises by 64 MB, naming the pool and
    consumer, and records the peak.
    - `sqllogictest`: a `CountingAllocator` global allocator, with every
    test file's pool wrapped against one process-wide tracker. The peak
    drift is printed at the end of the run. Disable it with `--memory-drift
    false`.
    
    Test files run concurrently, so the comparison is process-wide rather
    than per file. Files that `SET datafusion.runtime.memory_limit` replace
    their pool and drop out of the reserved total (6 of 504 files).
    
    Example output from a full local run:
    
    ```
    Peak memory drift: drift=84.3 MB allocated=85.4 MB reserved=1176.3 KB pool=order.slt consumer=ExternalSorterMerge[0]
    ```
    
    The allocator batches its counts per thread, so SLT runtime is unchanged
    locally (7s both with and without it).
    
    ## Are these changes tested?
    
    Yes. There are unit tests for the tracker and pool, plus a doc test. I
    also ran the full SLT suite locally with drift logging on and off.
    
    ## Are there any user-facing changes?
    
    New public types in `datafusion_execution::memory_pool`. No changes to
    existing APIs.
---
 datafusion/execution/src/memory_pool/drift.rs      | 392 +++++++++++++++++++++
 datafusion/execution/src/memory_pool/mod.rs        |   2 +
 .../execution/src/memory_pool/peak_recording.rs    |  36 +-
 datafusion/sqllogictest/README.md                  |  39 ++
 datafusion/sqllogictest/bin/sqllogictests.rs       |  41 +++
 .../src/engines/datafusion_engine/runner.rs        |   2 +
 .../runner.rs                                      |   2 +
 datafusion/sqllogictest/src/lib.rs                 |   5 +
 datafusion/sqllogictest/src/memory_drift.rs        | 181 ++++++++++
 datafusion/sqllogictest/src/test_context.rs        |  12 +-
 10 files changed, 706 insertions(+), 6 deletions(-)

diff --git a/datafusion/execution/src/memory_pool/drift.rs b/datafusion/execution/src/memory_pool/drift.rs
new file mode 100644
index 0000000000..ded3f7343f
--- /dev/null
+++ b/datafusion/execution/src/memory_pool/drift.rs
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Logs the drift between what [`MemoryPool`]s have reserved and what the
+//! process has actually allocated.
+//!
+//! [`MemoryPool`] accounting is voluntary: only allocations that an operator
+//! explicitly reserves are counted. Anything else (in-flight batches, kernel
+//! scratch space, untracked buffers) is invisible to the pool, so a process
+//! can run out of memory while its pool reports plenty of headroom. See
+//! <https://github.com/apache/datafusion/issues/25650>.
+//!
+//! This module only observes. Nothing here changes how memory is granted or
+//! limited; it logs the gap so that operators which under-report can be found.
+//!
+//! DataFusion is a library and does not choose the global allocator, so the
+//! allocated byte count is supplied by the caller, e.g. from a counting
+//! [`GlobalAlloc`](std::alloc::GlobalAlloc) wrapper or allocator statistics.
+//!
+//! [`MemoryPool`]: super::MemoryPool
+
+use std::{
+    fmt::{Debug, Display, Formatter},
+    sync::{
+        Arc,
+        atomic::{AtomicIsize, AtomicUsize, Ordering},
+    },
+};
+
+use datafusion_common::human_readable_size;
+use parking_lot::Mutex;
+
+/// Returns the number of bytes currently allocated by the process.
+///
+/// This is called on every reservation change (`grow`, `try_grow` and
+/// `shrink`) of every pool that reports to the tracker, so it must be cheap,
+/// e.g. a single atomic load. Do not read allocator statistics that need a
+/// refresh on each call (such as jemalloc's `epoch`).
+pub type AllocatedBytesFn = Arc<dyn Fn() -> usize + Send + Sync>;
+
+/// Default rise in drift, in bytes, needed before another line is logged.
+pub const DEFAULT_DRIFT_LOG_THRESHOLD: usize = 64 * 1024 * 1024;
+
+/// Compares allocated bytes against the total reserved by every pool that
+/// reports to it (see [`PeakRecordingPool::with_drift_tracker`]).
+///
+/// [`PeakRecordingPool::with_drift_tracker`]: super::PeakRecordingPool::with_drift_tracker
+///
+/// A single tracker can be shared by many pools, e.g. one pool per
+/// `SessionContext` in a process running several at once. The reserved total
+/// is then summed across all of them, which is what has to be compared with a
+/// process-wide allocated byte count.
+///
+/// Drift is `allocated - reserved`. A line is logged at `info` level each time
+/// drift rises by at least the log threshold, naming the pool and consumer
+/// whose reservation change triggered the check.
+///
+/// # Example
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use datafusion_execution::memory_pool::{
+/// #     MemoryConsumer, MemoryDriftTracker, MemoryPool, PeakRecordingPool, UnboundedMemoryPool,
+/// # };
+/// // A real caller would read a counting allocator or allocator stats here.
+/// let tracker = Arc::new(MemoryDriftTracker::new(Arc::new(|| 10_000)));
+/// let pool: Arc<dyn MemoryPool> = Arc::new(
+///     PeakRecordingPool::new(Arc::new(UnboundedMemoryPool::default()))
+///         .with_drift_tracker(Arc::clone(&tracker), "example"),
+/// );
+///
+/// let reservation = MemoryConsumer::new("op").register(&pool);
+/// reservation.grow(4_000);
+///
+/// assert_eq!(tracker.reserved(), 4_000);
+/// assert_eq!(tracker.peak_drift().unwrap().drift, 6_000);
+/// ```
+pub struct MemoryDriftTracker {
+    allocated: AllocatedBytesFn,
+    log_threshold: usize,
+    /// Total reserved across every pool reporting to this tracker.
+    reserved: AtomicUsize,
+    /// Positive drift at the time of the last logged line.
+    last_logged: AtomicIsize,
+    /// Largest drift seen.
+    peak_drift: AtomicIsize,
+    /// Where the largest drift was seen.
+    peak: Mutex<Option<DriftSample>>,
+}
+
+/// One observation of drift, recorded by [`MemoryDriftTracker`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct DriftSample {
+    /// Label of the pool whose reservation change took this sample, or the source passed to [`MemoryDriftTracker::sample`]. When
+    /// several pools share a tracker, the untracked memory can come from any
+    /// of them.
+    pub pool: String,
+    /// Consumer whose reservation change took this sample (empty for
+    /// [`MemoryDriftTracker::sample`]). This shows when drift was sampled,
+    /// not what caused it.
+    pub consumer: String,
+    /// Bytes reserved across all pools reporting to the tracker.
+    pub reserved: usize,
+    /// Bytes allocated, as reported by the tracker's [`AllocatedBytesFn`].
+    pub allocated: usize,
+    /// `allocated - reserved`.
+    pub drift: isize,
+}
+
+impl Display for DriftSample {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        let drift = if self.drift < 0 {
+            format!("-{}", human_readable_size(self.drift.unsigned_abs()))
+        } else {
+            human_readable_size(self.drift as usize)
+        };
+        write!(
+            f,
+            "drift={drift} allocated={} reserved={} pool={} consumer={}",
+            human_readable_size(self.allocated),
+            human_readable_size(self.reserved),
+            self.pool,
+            self.consumer,
+        )
+    }
+}
+
+impl MemoryDriftTracker {
+    /// Create a tracker that reads allocated bytes from `allocated`, logging
+    /// every [`DEFAULT_DRIFT_LOG_THRESHOLD`] of drift.
+    pub fn new(allocated: AllocatedBytesFn) -> Self {
+        Self {
+            allocated,
+            log_threshold: DEFAULT_DRIFT_LOG_THRESHOLD,
+            reserved: AtomicUsize::new(0),
+            last_logged: AtomicIsize::new(0),
+            peak_drift: AtomicIsize::new(isize::MIN),
+            peak: Mutex::new(None),
+        }
+    }
+
+    /// Log a line each time drift rises by `log_threshold` bytes.
+    pub fn with_log_threshold(mut self, log_threshold: usize) -> Self {
+        self.log_threshold = log_threshold;
+        self
+    }
+
+    /// Bytes currently reserved across all pools reporting to this tracker.
+    pub fn reserved(&self) -> usize {
+        self.reserved.load(Ordering::Relaxed)
+    }
+
+    /// The largest drift seen so far, if any reservation has been made.
+    pub fn peak_drift(&self) -> Option<DriftSample> {
+        self.peak.lock().clone()
+    }
+
+    /// Compare the current allocated bytes with the reserved total now,
+    /// without a reservation change.
+    ///
+    /// Drift is otherwise only sampled when a reservation changes, so memory
+    /// allocated by code that reserves little can go unseen until some other
+    /// reservation changes. Calling this periodically, e.g. from a timer,
+    /// closes that gap. `source` is recorded as the pool label.
+    pub fn sample(&self, source: &str) {
+        self.observe(source, "", self.reserved());
+    }
+
+    pub(super) fn grew(&self, pool: &str, consumer: &str, additional: usize) {
+        let reserved =
+            self.reserved.fetch_add(additional, Ordering::Relaxed) + additional;
+        self.observe(pool, consumer, reserved);
+    }
+
+    pub(super) fn shrank(&self, pool: &str, consumer: &str, shrink: usize) {
+        let reserved = self.reserved.fetch_sub(shrink, Ordering::Relaxed) - shrink;
+        self.observe(pool, consumer, reserved);
+    }
+
+    /// Returns `true` if a line was logged.
+    fn observe(&self, pool: &str, consumer: &str, reserved: usize) -> bool {
+        let allocated = (self.allocated)();
+        let drift = allocated as isize - reserved as isize;
+
+        let sample = || DriftSample {
+            pool: pool.to_string(),
+            consumer: consumer.to_string(),
+            reserved,
+            allocated,
+            drift,
+        };
+
+        // Lock-free check first so the lock is only taken for a new peak.
+        if self.peak_drift.fetch_max(drift, Ordering::Relaxed) < drift {
+            let mut peak = self.peak.lock();
+            if peak.as_ref().is_none_or(|p| drift > p.drift) {
+                *peak = Some(sample());
+            }
+        }
+
+        // Only rising positive drift is logged: that is untracked memory, which
+        // is what leads to OOM kills. Negative drift (e.g. an operator
+        // reserving ahead of allocating) counts as zero. Falling drift quietly
+        // lowers the baseline so the next rise is seen.
+        let untracked = drift.max(0);
+        let last = self.last_logged.load(Ordering::Relaxed);
+        let rose = untracked >= last.saturating_add_unsigned(self.log_threshold);
+        if !rose && untracked >= last {
+            return false;
+        }
+        let updated = self
+            .last_logged
+            .compare_exchange(last, untracked, Ordering::Relaxed, Ordering::Relaxed)
+            .is_ok();
+        let logged = updated && rose;
+        if logged {
+            log::info!("memory drift: {}", sample());
+        }
+        logged
+    }
+}
+
+impl Debug for MemoryDriftTracker {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("MemoryDriftTracker")
+            .field("log_threshold", &self.log_threshold)
+            .field("reserved", &self.reserved())
+            .field("peak", &self.peak_drift())
+            .finish()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::memory_pool::{
+        MemoryConsumer, MemoryPool, PeakRecordingPool, UnboundedMemoryPool,
+    };
+
+    use super::*;
+
+    /// A tracker whose allocated byte count is set by the test.
+    fn tracker(log_threshold: usize) -> (Arc<AtomicUsize>, Arc<MemoryDriftTracker>) {
+        let allocated = Arc::new(AtomicUsize::new(0));
+        let source = Arc::clone(&allocated);
+        let tracker =
+            MemoryDriftTracker::new(Arc::new(move || source.load(Ordering::Relaxed)))
+                .with_log_threshold(log_threshold);
+        (allocated, Arc::new(tracker))
+    }
+
+    fn pool(tracker: &Arc<MemoryDriftTracker>, label: &str) -> Arc<dyn MemoryPool> {
+        Arc::new(
+            PeakRecordingPool::new(Arc::new(UnboundedMemoryPool::default()))
+                .with_drift_tracker(Arc::clone(tracker), label),
+        )
+    }
+
+    #[test]
+    fn records_peak_drift_and_where_it_happened() {
+        let (allocated, tracker) = tracker(DEFAULT_DRIFT_LOG_THRESHOLD);
+        let pool = pool(&tracker, "q1");
+
+        let a = MemoryConsumer::new("a").register(&pool);
+        let b = MemoryConsumer::new("b").register(&pool);
+
+        allocated.store(1000, Ordering::Relaxed);
+        a.grow(800);
+        allocated.store(5000, Ordering::Relaxed);
+        b.grow(200);
+        allocated.store(1500, Ordering::Relaxed);
+        a.shrink(800);
+
+        assert_eq!(tracker.reserved(), 200);
+        assert_eq!(
+            tracker.peak_drift(),
+            Some(DriftSample {
+                pool: "q1".to_string(),
+                consumer: "b".to_string(),
+                reserved: 1000,
+                allocated: 5000,
+                drift: 4000,
+            })
+        );
+    }
+
+    #[test]
+    fn sums_reservations_across_pools_sharing_a_tracker() {
+        let (allocated, tracker) = tracker(DEFAULT_DRIFT_LOG_THRESHOLD);
+        let one = pool(&tracker, "one");
+        let two = pool(&tracker, "two");
+
+        allocated.store(1000, Ordering::Relaxed);
+        let a = MemoryConsumer::new("a").register(&one);
+        let b = MemoryConsumer::new("b").register(&two);
+        a.grow(300);
+        b.grow(400);
+
+        // Drift is measured against the sum, so the second pool's growth
+        // shrinks it rather than starting from zero.
+        assert_eq!(tracker.reserved(), 700);
+        let peak = tracker.peak_drift().unwrap();
+        assert_eq!(peak.pool, "one");
+        assert_eq!(peak.drift, 700);
+
+        drop(a);
+        drop(b);
+        assert_eq!(tracker.reserved(), 0);
+    }
+
+    #[test]
+    fn drift_can_be_negative() {
+        let (_allocated, tracker) = tracker(DEFAULT_DRIFT_LOG_THRESHOLD);
+        let pool = pool(&tracker, "over-reserved");
+
+        let reservation = MemoryConsumer::new("a").register(&pool);
+        reservation.grow(2048);
+
+        let peak = tracker.peak_drift().unwrap();
+        assert_eq!(peak.drift, -2048);
+        assert_eq!(
+            peak.to_string(),
+            "drift=-2.0 KB allocated=0.0 B reserved=2.0 KB pool=over-reserved consumer=a"
+        );
+    }
+
+    #[test]
+    fn logs_each_rise_by_the_threshold_and_rearms_after_a_fall() {
+        let (allocated, tracker) = tracker(1000);
+        let observe = |bytes: usize, reserved: usize| {
+            allocated.store(bytes, Ordering::Relaxed);
+            tracker.observe("p", "c", reserved)
+        };
+
+        // Negative drift counts as zero, so the baseline stays at zero.
+        assert!(!observe(0, 5000));
+        // Below the threshold: nothing logged.
+        assert!(!observe(999, 0));
+        // Reaching the threshold logs and moves the baseline to 1000.
+        assert!(observe(1000, 0));
+        assert!(!observe(1999, 0));
+        assert!(observe(2000, 0));
+        // Falling drift is not logged but lowers the baseline to 500...
+        assert!(!observe(500, 0));
+        // ...so a rise of the threshold from there logs again.
+        assert!(!observe(1499, 0));
+        assert!(observe(1500, 0));
+    }
+
+    #[test]
+    fn a_threshold_above_isize_max_never_logs() {
+        let (allocated, tracker) = tracker(usize::MAX);
+
+        allocated.store(1 << 40, Ordering::Relaxed);
+        assert!(!tracker.observe("p", "c", 0));
+    }
+
+    #[test]
+    fn sample_observes_without_a_reservation_change() {
+        let (allocated, tracker) = tracker(DEFAULT_DRIFT_LOG_THRESHOLD);
+        let pool = pool(&tracker, "q1");
+        let reservation = MemoryConsumer::new("a").register(&pool);
+        reservation.grow(100);
+
+        allocated.store(5000, Ordering::Relaxed);
+        tracker.sample("timer");
+
+        assert_eq!(
+            tracker.peak_drift(),
+            Some(DriftSample {
+                pool: "timer".to_string(),
+                consumer: String::new(),
+                reserved: 100,
+                allocated: 5000,
+                drift: 4900,
+            })
+        );
+    }
+}
diff --git a/datafusion/execution/src/memory_pool/mod.rs b/datafusion/execution/src/memory_pool/mod.rs
index 5df64ba47b..09cef6ddd4 100644
--- a/datafusion/execution/src/memory_pool/mod.rs
+++ b/datafusion/execution/src/memory_pool/mod.rs
@@ -24,6 +24,7 @@ use std::fmt::Display;
 use std::hash::{Hash, Hasher};
 use std::{cmp::Ordering, sync::Arc, sync::atomic};
 
+mod drift;
 mod merge_memory_pool;
 mod peak_recording;
 mod pool;
@@ -38,6 +39,7 @@ pub mod proxy {
 pub use datafusion_common::{
     human_readable_count, human_readable_duration, human_readable_size, units,
 };
+pub use drift::*;
 pub use merge_memory_pool::{MergeMemoryPool, WorkspaceLoan};
 pub use peak_recording::*;
 pub use pool::*;
diff --git a/datafusion/execution/src/memory_pool/peak_recording.rs b/datafusion/execution/src/memory_pool/peak_recording.rs
index b407cc0eaf..3c916b5ac7 100644
--- a/datafusion/execution/src/memory_pool/peak_recording.rs
+++ b/datafusion/execution/src/memory_pool/peak_recording.rs
@@ -33,6 +33,10 @@
 //! This is measurement only: nothing here enforces a relationship between the
 //! two numbers.
 //!
+//! The same wrapper can also report every reservation change to a
+//! [`MemoryDriftTracker`] (see [`PeakRecordingPool::with_drift_tracker`]),
+//! which compares reservations with the bytes actually allocated.
+//!
 //! What lands in the peak is whatever the pool accounts for, so this follows
 //! the accounting rather than fixing it in place. Arrow-side reservations made
 //! through `ArrowMemoryPool` are included, because that adapter grows a
@@ -47,7 +51,9 @@ use std::{
     },
 };
 
-use super::{MemoryConsumer, MemoryLimit, MemoryPool, MemoryReservation};
+use super::{
+    MemoryConsumer, MemoryDriftTracker, MemoryLimit, MemoryPool, MemoryReservation,
+};
 use datafusion_common::Result;
 
 /// Wraps a [`MemoryPool`], recording the high-water mark of
@@ -92,6 +98,8 @@ pub struct PeakRecordingPool {
     peak: AtomicUsize,
     /// High-water mark since this pool was created. Never reset.
     max: AtomicUsize,
+    /// Tracker every reservation change is reported to, and this pool's label.
+    drift: Option<(Arc<MemoryDriftTracker>, String)>,
 }
 
 impl PeakRecordingPool {
@@ -105,9 +113,22 @@ impl PeakRecordingPool {
             reserved: AtomicUsize::new(0),
             peak: AtomicUsize::new(0),
             max: AtomicUsize::new(0),
+            drift: None,
         }
     }
 
+    /// Also report every reservation change to `tracker`. `label` identifies
+    /// this pool in drift log lines, which matters when several pools share
+    /// one tracker.
+    pub fn with_drift_tracker(
+        mut self,
+        tracker: Arc<MemoryDriftTracker>,
+        label: impl Into<String>,
+    ) -> Self {
+        self.drift = Some((tracker, label.into()));
+        self
+    }
+
     /// The recorder installed as `pool`, if there is one.
     ///
     /// Returns `None` whenever a benchmark runs without a memory limit, since
@@ -152,11 +173,14 @@ impl PeakRecordingPool {
     /// The total stays exact because the trait grants exactly what is asked
     /// for — `grow` is infallible and `try_grow` either grants `additional` or
     /// returns an error, leaving the reservation untouched.
-    fn record(&self, additional: usize) {
+    fn record(&self, reservation: &MemoryReservation, additional: usize) {
         let reserved =
             self.reserved.fetch_add(additional, Ordering::Relaxed) + additional;
         self.peak.fetch_max(reserved, Ordering::Relaxed);
         self.max.fetch_max(reserved, Ordering::Relaxed);
+        if let Some((tracker, label)) = &self.drift {
+            tracker.grew(label, reservation.consumer().name(), additional);
+        }
     }
 }
 
@@ -166,6 +190,7 @@ impl Debug for PeakRecordingPool {
             .field("inner", &self.inner)
             .field("peak", &self.peak_reserved())
             .field("max", &self.max_reserved())
+            .field("drift_label", &self.drift.as_ref().map(|(_, label)| label))
             .finish()
     }
 }
@@ -193,17 +218,20 @@ impl MemoryPool for PeakRecordingPool {
 
     fn grow(&self, reservation: &MemoryReservation, additional: usize) {
         self.inner.grow(reservation, additional);
-        self.record(additional);
+        self.record(reservation, additional);
     }
 
     fn shrink(&self, reservation: &MemoryReservation, shrink: usize) {
         self.inner.shrink(reservation, shrink);
         self.reserved.fetch_sub(shrink, Ordering::Relaxed);
+        if let Some((tracker, label)) = &self.drift {
+            tracker.shrank(label, reservation.consumer().name(), shrink);
+        }
     }
 
     fn try_grow(&self, reservation: &MemoryReservation, additional: usize) -> Result<()> {
         self.inner.try_grow(reservation, additional)?;
-        self.record(additional);
+        self.record(reservation, additional);
         Ok(())
     }
 
diff --git a/datafusion/sqllogictest/README.md b/datafusion/sqllogictest/README.md
index 13ef797dfc..ad6c552f75 100644
--- a/datafusion/sqllogictest/README.md
+++ b/datafusion/sqllogictest/README.md
@@ -101,6 +101,45 @@ SLT_TIMING_SUMMARY=1 cargo test --test sqllogictests
 SLT_TIMING_DEBUG_SLOW_FILES=1 cargo test --test sqllogictests
 ```
 
+### Memory drift
+
+The runner compares the bytes reserved in `MemoryPool`s with the bytes actually
+allocated, to find operators whose memory is not tracked by the pool (see
+[#25650](https://github.com/apache/datafusion/issues/25650)). It is enabled by
+default and prints the largest drift seen at the end of the run. It only logs
+and never fails a test.
+
+Test files run concurrently by default, so the comparison is process-wide:
+allocated bytes across the whole process against reservations summed across all
+files. In this mode, the file in the output can be wrong. To attribute drift to
+one file, run one file at a time:
+
+```shell
+cargo test --test sqllogictests -- --test-threads 1
+```
+
+In both modes, the consumer in the output is the reservation change that took
+the sample, not necessarily the code that allocated the untracked memory. Most
+drift at this scale is memory that `MemoryPool` does not track by design (for
+example, in-flight batches and Parquet read buffers), so a large drift is a lead
+to investigate, not necessarily a bug.
+
+```shell
+# Log each 64 MB rise in drift, with the file and consumer that took the sample
+RUST_LOG=datafusion_execution::memory_pool=info cargo test --test sqllogictests
+```
+
+```shell
+# Log each 4 MB rise instead (also settable via SLT_MEMORY_DRIFT_LOG_THRESHOLD)
+RUST_LOG=datafusion_execution::memory_pool=info cargo test --test sqllogictests -- \
+  --test-threads 1 --memory-drift-log-threshold 4194304
+```
+
+```shell
+# Disable (allocations are still counted, but pools are not wrapped)
+cargo test --test sqllogictests -- --memory-drift false
+```
+
 ## Cookbook: Adding Tests
 
 1. Add queries
diff --git a/datafusion/sqllogictest/bin/sqllogictests.rs b/datafusion/sqllogictest/bin/sqllogictests.rs
index 71573c5e53..c64864cb77 100644
--- a/datafusion/sqllogictest/bin/sqllogictests.rs
+++ b/datafusion/sqllogictest/bin/sqllogictests.rs
@@ -19,9 +19,14 @@ use clap::{ColorChoice, Parser};
 use datafusion::common::instant::Instant;
 use datafusion::common::utils::get_available_parallelism;
 use datafusion::common::{DataFusionError, Result, exec_datafusion_err, exec_err};
+use datafusion::execution::memory_pool::DEFAULT_DRIFT_LOG_THRESHOLD;
 #[cfg(feature = "substrait")]
 use datafusion_sqllogictest::DataFusionSubstraitRoundTrip;
 use datafusion_sqllogictest::TestFile;
+use datafusion_sqllogictest::{
+    CountingAllocator, enable_memory_drift_logging, flush_thread_allocations,
+    memory_drift_tracker,
+};
 use datafusion_sqllogictest::{
     CurrentlyExecutingSqlTracker, DFColumnType, DataFusion, Filter, TestContext,
     df_value_validator, read_dir_recursive, run_each_configuration, setup_scratch_dir,
@@ -57,6 +62,9 @@ use std::time::Duration;
 #[cfg(feature = "postgres")]
 mod postgres_container;
 
+#[global_allocator]
+static ALLOC: CountingAllocator = CountingAllocator;
+
 const TEST_DIRECTORY: &str = "test_files/";
 const DATAFUSION_TESTING_TEST_DIRECTORY: &str = "../../datafusion-testing/data/";
 const PG_COMPAT_FILE_PREFIX: &str = "pg_compat_";
@@ -88,6 +96,7 @@ fn config_change_result(
 pub fn main() -> Result<()> {
     tokio::runtime::Builder::new_multi_thread()
         .enable_all()
+        .on_thread_stop(flush_thread_allocations)
         .build()?
         .block_on(run_tests())
 }
@@ -139,6 +148,10 @@ async fn run_tests() -> Result<()> {
 
     options.warn_on_ignored();
 
+    if options.memory_drift {
+        enable_memory_drift_logging(options.memory_drift_log_threshold);
+    }
+
     // Print parallelism info for debugging CI performance
     eprintln!(
         "Running with {} test threads (available parallelism: {})",
@@ -386,6 +399,16 @@ async fn run_tests() -> Result<()> {
         HumanDuration(start.elapsed())
     ))?;
 
+    if let Some(peak) = memory_drift_tracker().and_then(|t| t.peak_drift()) {
+        eprintln!("Peak memory drift: {peak}");
+        if options.test_threads > 1 {
+            eprintln!(
+                "Test files ran concurrently, so the file and consumer above can be wrong. \
+                 Run with --test-threads 1 to attribute drift to one file."
+            );
+        }
+    }
+
     #[cfg(feature = "postgres")]
     terminate_postgres_container().await?;
 
@@ -1106,6 +1129,24 @@ struct Options {
     )]
     timing_summary: bool,
 
+    #[clap(
+        long,
+        env = "SLT_MEMORY_DRIFT",
+        default_value_t = true,
+        action = clap::ArgAction::Set,
+        help = "Log drift between MemoryPool reservations and allocated bytes (RUST_LOG=datafusion_execution::memory_pool=info to log each rise of --memory-drift-log-threshold bytes)"
+    )]
+    memory_drift: bool,
+
+    #[clap(
+        long,
+        env = "SLT_MEMORY_DRIFT_LOG_THRESHOLD",
+        value_name = "BYTES",
+        default_value_t = DEFAULT_DRIFT_LOG_THRESHOLD,
+        help = "Rise in memory drift, in bytes, needed before another line is logged"
+    )]
+    memory_drift_log_threshold: usize,
+
     #[clap(
         long,
         value_name = "MODE",
diff --git a/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs b/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
index 470efa6f4f..67d4c26a54 100644
--- a/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
+++ b/datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs
@@ -24,6 +24,7 @@ use super::{DFSqlLogicTestError, error::Result, normalize};
 use crate::engines::currently_executed_sql::CurrentlyExecutingSqlTracker;
 use crate::engines::output::{DFColumnType, DFOutput};
 use crate::is_spark_path;
+use crate::memory_drift::rewrap_replaced_pool;
 use arrow::record_batch::RecordBatch;
 use async_trait::async_trait;
 use datafusion::physical_plan::common::collect;
@@ -156,6 +157,7 @@ impl sqllogictest::AsyncDB for DataFusion {
         let start = Instant::now();
         let result = run_query(&self.ctx, is_spark_path(&self.relative_path), sql).await;
         let duration = start.elapsed();
+        rewrap_replaced_pool(&self.ctx, &self.relative_path.display().to_string());
 
         self.currently_executing_sql_tracker.remove_sql(tracked_sql);
 
diff --git a/datafusion/sqllogictest/src/engines/datafusion_substrait_roundtrip_engine/runner.rs b/datafusion/sqllogictest/src/engines/datafusion_substrait_roundtrip_engine/runner.rs
index 90ad6646dc..02580eec91 100644
--- a/datafusion/sqllogictest/src/engines/datafusion_substrait_roundtrip_engine/runner.rs
+++ b/datafusion/sqllogictest/src/engines/datafusion_substrait_roundtrip_engine/runner.rs
@@ -21,6 +21,7 @@ use std::{path::PathBuf, time::Duration};
 use crate::engines::currently_executed_sql::CurrentlyExecutingSqlTracker;
 use crate::engines::datafusion_engine::Result;
 use crate::engines::output::{DFColumnType, DFOutput};
+use crate::memory_drift::rewrap_replaced_pool;
 use crate::{DFSqlLogicTestError, convert_batches, convert_schema_to_types};
 use arrow::record_batch::RecordBatch;
 use async_trait::async_trait;
@@ -102,6 +103,7 @@ impl sqllogictest::AsyncDB for DataFusionSubstraitRoundTrip {
         let start = Instant::now();
         let result = run_query_substrait_round_trip(&self.ctx, sql).await;
         let duration = start.elapsed();
+        rewrap_replaced_pool(&self.ctx, &self.relative_path.display().to_string());
 
         self.currently_executing_sql_tracker.remove_sql(tracked_sql);
 
diff --git a/datafusion/sqllogictest/src/lib.rs b/datafusion/sqllogictest/src/lib.rs
index 926b1fb1e8..a397763933 100644
--- a/datafusion/sqllogictest/src/lib.rs
+++ b/datafusion/sqllogictest/src/lib.rs
@@ -44,11 +44,16 @@ pub use engines::convert_schema_to_types;
 pub use engines::Postgres;
 
 mod filters;
+mod memory_drift;
 mod test_context;
 mod util;
 
 pub use config_matrix::{TestConfiguration, run_each_configuration, test_configurations};
 pub use filters::*;
+pub use memory_drift::{
+    CountingAllocator, enable_memory_drift_logging, flush_thread_allocations,
+    memory_drift_tracker,
+};
 pub use test_context::TestContext;
 pub use test_file::TestFile;
 pub use util::*;
diff --git a/datafusion/sqllogictest/src/memory_drift.rs b/datafusion/sqllogictest/src/memory_drift.rs
new file mode 100644
index 0000000000..2aff6499bf
--- /dev/null
+++ b/datafusion/sqllogictest/src/memory_drift.rs
@@ -0,0 +1,181 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Logs drift between `MemoryPool` reservations and actual allocations while
+//! running sqllogictests. See <https://github.com/apache/datafusion/issues/25650>.
+//!
+//! Test files run concurrently in one process, so allocations cannot be split
+//! per file. Instead every file's pool reports to one process-wide
+//! [`MemoryDriftTracker`], which compares the sum of all reservations with the
+//! bytes counted by [`CountingAllocator`].
+//!
+//! Files that `SET datafusion.runtime.memory_limit` replace their pool; the
+//! runner wraps the replacement too (see [`rewrap_replaced_pool`]).
+//!
+//! This only logs. It never fails a test.
+
+use std::{
+    alloc::{GlobalAlloc, Layout, System},
+    cell::Cell,
+    sync::{
+        Arc, OnceLock,
+        atomic::{AtomicIsize, Ordering},
+    },
+};
+
+use datafusion::execution::SessionStateBuilder;
+use datafusion::execution::memory_pool::{
+    MemoryDriftTracker, MemoryPool, PeakRecordingPool,
+};
+use datafusion::execution::runtime_env::RuntimeEnvBuilder;
+use datafusion::prelude::SessionContext;
+
+static ALLOCATED: AtomicIsize = AtomicIsize::new(0);
+static TRACKER: OnceLock<Arc<MemoryDriftTracker>> = OnceLock::new();
+
+/// A [`GlobalAlloc`] that counts the bytes currently allocated through it,
+/// delegating the allocation itself to [`System`].
+///
+/// Counts requested sizes, so allocator overhead and memory retained by the
+/// allocator are not included. Counting starts with the process, so memory
+/// freed later was always counted when it was allocated.
+pub struct CountingAllocator;
+
+/// Per-thread count is flushed to [`ALLOCATED`] once it moves this far, so
+/// threads do not contend on one atomic for every allocation. The global count
+/// is therefore off by up to `FLUSH_BYTES` per live thread. Tokio threads
+/// flush the rest when they stop (see [`flush_thread_allocations`]); the
+/// unflushed count of any other thread that exits is lost.
+const FLUSH_BYTES: isize = 256 * 1024;
+
+thread_local! {
+    // `const` with no destructor, so accessing it never allocates.
+    static UNFLUSHED: Cell<isize> = const { Cell::new(0) };
+}
+
+fn count(delta: isize) {
+    // `try_with` because this can run while the thread is being torn down.
+    let _ = UNFLUSHED.try_with(|unflushed| {
+        let pending = unflushed.get() + delta;
+        if pending.abs() >= FLUSH_BYTES {
+            ALLOCATED.fetch_add(pending, Ordering::Relaxed);
+            unflushed.set(0);
+        } else {
+            unflushed.set(pending);
+        }
+    });
+}
+
+/// Flush this thread's unflushed count into the global count.
+///
+/// Registered as the Tokio runtime's `on_thread_stop` hook, which runs just
+/// before a worker or blocking thread exits, outside the allocator.
+pub fn flush_thread_allocations() {
+    let _ = UNFLUSHED.try_with(|unflushed| {
+        ALLOCATED.fetch_add(unflushed.replace(0), Ordering::Relaxed);
+    });
+}
+
+// SAFETY: every call is forwarded unchanged to `System`, which upholds the
+// `GlobalAlloc` contract. Counting has no effect on the returned memory.
+unsafe impl GlobalAlloc for CountingAllocator {
+    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
+        let ptr = unsafe { System.alloc(layout) };
+        if !ptr.is_null() {
+            count(layout.size() as isize);
+        }
+        ptr
+    }
+
+    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
+        let ptr = unsafe { System.alloc_zeroed(layout) };
+        if !ptr.is_null() {
+            count(layout.size() as isize);
+        }
+        ptr
+    }
+
+    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
+        unsafe { System.dealloc(ptr, layout) };
+        count(-(layout.size() as isize));
+    }
+
+    unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
+        let new_ptr = unsafe { System.realloc(ptr, layout, new_size) };
+        if !new_ptr.is_null() {
+            count(new_size as isize - layout.size() as isize);
+        }
+        new_ptr
+    }
+}
+
+/// Bytes currently allocated through [`CountingAllocator`], clamped at zero
+/// because counts lost with exiting threads can bias it either way.
+fn allocated_bytes() -> usize {
+    ALLOCATED.load(Ordering::Relaxed).max(0) as usize
+}
+
+/// Wrap the memory pool of every test file created from now on, logging each
+/// rise in drift of `log_threshold` bytes. Only meaningful if
+/// [`CountingAllocator`] is the global allocator.
+pub fn enable_memory_drift_logging(log_threshold: usize) {
+    TRACKER.get_or_init(|| {
+        Arc::new(
+            MemoryDriftTracker::new(Arc::new(allocated_bytes))
+                .with_log_threshold(log_threshold),
+        )
+    });
+}
+
+/// The process-wide tracker, if [`enable_memory_drift_logging`] was called.
+pub fn memory_drift_tracker() -> Option<&'static Arc<MemoryDriftTracker>> {
+    TRACKER.get()
+}
+
+/// Wrap `pool` so its reservations are reported to the process-wide tracker,
+/// or return it unchanged if drift logging is not enabled.
+pub fn wrap_pool(pool: Arc<dyn MemoryPool>, label: &str) -> Arc<dyn MemoryPool> {
+    match memory_drift_tracker() {
+        Some(tracker) => Arc::new(
+            PeakRecordingPool::new(pool).with_drift_tracker(Arc::clone(tracker), label),
+        ),
+        None => pool,
+    }
+}
+
+/// Wrap the memory pool of `ctx` again if a statement replaced it, e.g.
+/// `SET datafusion.runtime.memory_limit`, so its reservations keep being
+/// counted. Does nothing if drift logging is not enabled.
+pub(crate) fn rewrap_replaced_pool(ctx: &SessionContext, label: &str) {
+    if memory_drift_tracker().is_none() {
+        return;
+    }
+    let state = ctx.state_ref();
+    let mut state = state.write();
+    let runtime = state.runtime_env();
+    if runtime.memory_pool.is::<PeakRecordingPool>() {
+        return;
+    }
+    let pool = wrap_pool(Arc::clone(&runtime.memory_pool), label);
+    let runtime = RuntimeEnvBuilder::from_runtime_env(runtime)
+        .with_memory_pool(pool)
+        .build_arc()
+        .expect("rebuilding an existing runtime succeeds");
+    *state = SessionStateBuilder::from(state.clone())
+        .with_runtime_env(runtime)
+        .build();
+}
diff --git a/datafusion/sqllogictest/src/test_context.rs b/datafusion/sqllogictest/src/test_context.rs
index 97d9d66867..bc2859aaa9 100644
--- a/datafusion/sqllogictest/src/test_context.rs
+++ b/datafusion/sqllogictest/src/test_context.rs
@@ -61,7 +61,8 @@ use range_partitioning::{
 use async_trait::async_trait;
 use datafusion::common::cast::as_float64_array;
 use datafusion::execution::SessionStateBuilder;
-use datafusion::execution::runtime_env::RuntimeEnv;
+use datafusion::execution::memory_pool::UnboundedMemoryPool;
+use datafusion::execution::runtime_env::RuntimeEnvBuilder;
 use datafusion::physical_plan::operator_statistics::StatisticsRegistry;
 use log::info;
 use sqlparser::ast;
@@ -110,7 +111,14 @@ impl TestContext {
         let config = SessionConfig::new()
             // hardcode target partitions so plans are deterministic
             .with_target_partitions(4);
-        let runtime = Arc::new(RuntimeEnv::default());
+        let pool = crate::memory_drift::wrap_pool(
+            Arc::new(UnboundedMemoryPool::default()),
+            &relative_path.display().to_string(),
+        );
+        let runtime = RuntimeEnvBuilder::new()
+            .with_memory_pool(pool)
+            .build_arc()
+            .expect("default runtime builds");
 
         let mut state_builder = SessionStateBuilder::new()
             .with_config(config)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org