You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/03 10:06:09 UTC

(datafusion) branch main updated: test: skip count distinct spill memory test under forced hash collisions (#24918)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git


The following commit(s) were added to refs/heads/main by this push:
     new 4a93adee06 test: skip count distinct spill memory test under forced hash collisions (#24918)
4a93adee06 is described below

commit 4a93adee064f63ae4d78fa92a39edf68c5dfb9f8
Author: Raz Luvaton <16...@users.noreply.github.com>
AuthorDate: Thu Sep 3 09:43:06 2026 +0000

    test: skip count distinct spill memory test under forced hash collisions (#24918)
    
    ## Which issue does this PR close?
    
    Follow-up to #24888, which broke the `cargo test hash collisions` CI job
    on main.
    
    ## Rationale for this change
    
    The memory limit test added in #24888 fails when built with
    `force_hash_collisions`. Every key hashes to the same value there, so
    the hash repartition sends all 64 groups to one final stage. That single
    table needs 5.3 MB against the test's 4 MB pool, and it has nothing
    reserved yet, so there is nothing to spill. It fails no matter how well
    the accumulator releases memory, which is what the test is actually
    about.
    
    I tried a few ways to keep it running under the feature first:
    
    - **Bigger limit for the collision build.** Below 5.3 MB it dies on that
    one state batch; at 6 MB and up nothing spills, so the unfixed
    accumulator passes too and the test asserts nothing. Nothing in between.
    - **Single partition, no repartition at all.** Same wall. With 64 groups
    the whole distinct state lives in 64 rows, so total state and one batch
    are the same 5.3 MB. Also 82s instead of 0.18s.
    - **More groups, to spread the state over more batches.** With every key
    in one hash bucket, interning goes quadratic: 4096 groups did not finish
    in 400s.
    - **More rows (800k), to make total state exceed one batch.** Fails even
    with the fix.
    
    They all hit the same thing: under forced collisions the total state and
    a single batch are the same size, and the pool would have to sit above
    one and below the other.
    
    ## What changes are included in this PR?
    
    The test and its helpers move into a module gated on `not(feature =
    "force_hash_collisions")`.
    
    ## What is the testing strategy for this PR?
    
    `cargo test -p datafusion --features force_hash_collisions --test
    core_integration count_distinct_releases` runs 0 tests. Without the
    feature it still runs and passes.
    
    ## Are there any user-facing changes?
    
    No.
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
 datafusion/core/tests/memory_limit/mod.rs | 173 ++++++++++++++++--------------
 1 file changed, 93 insertions(+), 80 deletions(-)

diff --git a/datafusion/core/tests/memory_limit/mod.rs b/datafusion/core/tests/memory_limit/mod.rs
index 39be33587f..9172e34a87 100644
--- a/datafusion/core/tests/memory_limit/mod.rs
+++ b/datafusion/core/tests/memory_limit/mod.rs
@@ -26,14 +26,11 @@ mod nlj_spill_unmatched;
 mod repartition_mem_limit;
 mod union_nullable_spill;
 mod view_spill_compaction;
-use arrow::array::{
-    ArrayRef, DictionaryArray, Int32Array, Int64Array, RecordBatch, StringViewArray,
-};
+use arrow::array::{ArrayRef, DictionaryArray, Int32Array, RecordBatch, StringViewArray};
 use arrow::compute::SortOptions;
 use arrow::datatypes::{Int32Type, SchemaRef};
 use arrow_schema::{DataType, Field, Schema};
 use datafusion::assert_batches_eq;
-use datafusion::assert_batches_sorted_eq;
 use datafusion::config::SpillCompression;
 use datafusion::datasource::memory::MemorySourceConfig;
 use datafusion::datasource::source::DataSourceExec;
@@ -128,83 +125,99 @@ async fn group_by_hash() {
         .await
 }
 
-/// `count(distinct)` over integers under a memory limit.
-///
-/// The integer distinct-count groups accumulator reports the capacity of its
-/// buffers in `size()`. After an aggregate stream emits all groups, either to
-/// emit partial state early or to spill, it resizes its reservation to the
-/// table's reported size and expects it to have shrunk. If the accumulator
-/// keeps its capacity, that resize is a grow against an exhausted pool and the
-/// query fails although everything was already written out.
-const COUNT_DISTINCT_ROWS: usize = 200_000;
-const COUNT_DISTINCT_GROUPS: i64 = 64;
-const COUNT_DISTINCT_BATCH_ROWS: usize = 8_192;
-
-/// Far below the distinct sets (200k values, several megabytes across the
-/// partial and final tables), far above the fixed cost of the stages. With the
-/// accumulator releasing its buffers the query passes from 2 MB upwards;
-/// without, it fails up to 4 MB with "Decreasing allocation after spilling
-/// should succeed" in the final stage or a failed emit in the partial stage.
-const COUNT_DISTINCT_MEMORY_LIMIT: usize = 4 * 1024 * 1024;
-
-/// `g` has 64 groups, `v` is unique, so every group holds 3125 distinct values.
-fn count_distinct_table() -> MemTable {
-    let schema = Arc::new(Schema::new(vec![
-        Field::new("g", DataType::Int64, false),
-        Field::new("v", DataType::Int64, false),
-    ]));
-    let batches = (0..COUNT_DISTINCT_ROWS)
-        .step_by(COUNT_DISTINCT_BATCH_ROWS)
-        .map(|start| {
-            let rows =
-                start..(start + COUNT_DISTINCT_BATCH_ROWS).min(COUNT_DISTINCT_ROWS);
-            RecordBatch::try_new(
-                Arc::clone(&schema),
-                vec![
-                    Arc::new(Int64Array::from_iter_values(
-                        rows.clone().map(|row| row as i64 % COUNT_DISTINCT_GROUPS),
-                    )),
-                    Arc::new(Int64Array::from_iter_values(rows.map(|row| row as i64))),
-                ],
-            )
-            .unwrap()
-        })
-        .collect();
-    MemTable::try_new(schema, vec![batches]).unwrap()
-}
-
-/// Four partial stages emit their state early and four hash-partitioned final
-/// stages spill; every one of them must see the accumulator memory drop after
-/// emitting all groups.
-#[tokio::test]
-async fn count_distinct_releases_memory_after_emitting_all() {
-    let runtime = RuntimeEnvBuilder::new()
-        .with_memory_limit(COUNT_DISTINCT_MEMORY_LIMIT, 1.0)
-        .with_disk_manager_builder(DiskManagerBuilder::default())
-        .build_arc()
-        .unwrap();
-    let config = SessionConfig::new().with_target_partitions(4);
-    let ctx = SessionContext::new_with_config_rt(config, runtime);
-    ctx.register_table("t", Arc::new(count_distinct_table()))
-        .unwrap();
+/// With `force_hash_collisions` every key hashes alike, so the hash
+/// repartitioning sends all groups to a single final stage, whose table then
+/// does not fit the limit however well memory is released. The limit is sized
+/// for the real distribution across four final stages, so this test is skipped
+/// under that feature.
+#[cfg(not(feature = "force_hash_collisions"))]
+mod count_distinct_spill {
+    use super::*;
+    use arrow::array::Int64Array;
+    use datafusion::assert_batches_sorted_eq;
+
+    /// `count(distinct)` over integers under a memory limit.
+    ///
+    /// The integer distinct-count groups accumulator reports the capacity of its
+    /// buffers in `size()`. After an aggregate stream emits all groups, either to
+    /// emit partial state early or to spill, it resizes its reservation to the
+    /// table's reported size and expects it to have shrunk. If the accumulator
+    /// keeps its capacity, that resize is a grow against an exhausted pool and the
+    /// query fails although everything was already written out.
+    const COUNT_DISTINCT_ROWS: usize = 200_000;
+    const COUNT_DISTINCT_GROUPS: i64 = 64;
+    const COUNT_DISTINCT_BATCH_ROWS: usize = 8_192;
+
+    /// Far below the distinct sets (200k values, several megabytes across the
+    /// partial and final tables), far above the fixed cost of the stages. With the
+    /// accumulator releasing its buffers the query passes from 2 MB upwards;
+    /// without, it fails up to 4 MB with "Decreasing allocation after spilling
+    /// should succeed" in the final stage or a failed emit in the partial stage.
+    const COUNT_DISTINCT_MEMORY_LIMIT: usize = 4 * 1024 * 1024;
+
+    /// `g` has 64 groups, `v` is unique, so every group holds 3125 distinct values.
+    fn count_distinct_table() -> MemTable {
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("g", DataType::Int64, false),
+            Field::new("v", DataType::Int64, false),
+        ]));
+        let batches = (0..COUNT_DISTINCT_ROWS)
+            .step_by(COUNT_DISTINCT_BATCH_ROWS)
+            .map(|start| {
+                let rows =
+                    start..(start + COUNT_DISTINCT_BATCH_ROWS).min(COUNT_DISTINCT_ROWS);
+                RecordBatch::try_new(
+                    Arc::clone(&schema),
+                    vec![
+                        Arc::new(Int64Array::from_iter_values(
+                            rows.clone().map(|row| row as i64 % COUNT_DISTINCT_GROUPS),
+                        )),
+                        Arc::new(Int64Array::from_iter_values(
+                            rows.map(|row| row as i64),
+                        )),
+                    ],
+                )
+                .unwrap()
+            })
+            .collect();
+        MemTable::try_new(schema, vec![batches]).unwrap()
+    }
 
-    let batches = ctx
-        .sql("select count(distinct v) as d, count(*) as n from t group by g")
-        .await
-        .unwrap()
-        .collect()
-        .await
-        .unwrap_or_else(|error| panic!("query failed under the memory limit: {error}"));
-
-    let per_group = (COUNT_DISTINCT_ROWS as i64 / COUNT_DISTINCT_GROUPS).to_string();
-    let row = format!("| {per_group} | {per_group} |");
-    let mut expected = vec!["+------+------+", "| d    | n    |", "+------+------+"];
-    expected.extend(std::iter::repeat_n(
-        row.as_str(),
-        COUNT_DISTINCT_GROUPS as usize,
-    ));
-    expected.push("+------+------+");
-    assert_batches_sorted_eq!(expected, &batches);
+    /// Four partial stages emit their state early and four hash-partitioned final
+    /// stages spill; every one of them must see the accumulator memory drop after
+    /// emitting all groups.
+    #[tokio::test]
+    async fn count_distinct_releases_memory_after_emitting_all() {
+        let runtime = RuntimeEnvBuilder::new()
+            .with_memory_limit(COUNT_DISTINCT_MEMORY_LIMIT, 1.0)
+            .with_disk_manager_builder(DiskManagerBuilder::default())
+            .build_arc()
+            .unwrap();
+        let config = SessionConfig::new().with_target_partitions(4);
+        let ctx = SessionContext::new_with_config_rt(config, runtime);
+        ctx.register_table("t", Arc::new(count_distinct_table()))
+            .unwrap();
+
+        let batches = ctx
+            .sql("select count(distinct v) as d, count(*) as n from t group by g")
+            .await
+            .unwrap()
+            .collect()
+            .await
+            .unwrap_or_else(|error| {
+                panic!("query failed under the memory limit: {error}")
+            });
+
+        let per_group = (COUNT_DISTINCT_ROWS as i64 / COUNT_DISTINCT_GROUPS).to_string();
+        let row = format!("| {per_group} | {per_group} |");
+        let mut expected = vec!["+------+------+", "| d    | n    |", "+------+------+"];
+        expected.extend(std::iter::repeat_n(
+            row.as_str(),
+            COUNT_DISTINCT_GROUPS as usize,
+        ));
+        expected.push("+------+------+");
+        assert_batches_sorted_eq!(expected, &batches);
+    }
 }
 
 #[tokio::test]


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org