You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2026/04/12 18:11:02 UTC

[PR] chore: add count distinct group benchmarks [datafusion]

coderfender opened a new pull request, #21575:
URL: https://github.com/apache/datafusion/pull/21575

   ## Which issue does this PR close?
   
   Add benchmarks for group accumulators to test :  https://github.com/apache/datafusion/pull/21561
   The implementation forks out based on `is_groups_accumulator_supported` function call. Once this is merged , we should be able to evaluate group accumulators on count distinct expr 
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#issuecomment-4232536196

   @Dandandan , I plan to add benches to help better evaluate group accumulators along with direct clickbench / TPCH queries for implementing group accumulator


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #21575:
URL: https://github.com/apache/datafusion/pull/21575


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3077240321


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   I agree that having a pre-optimization benchmark is useful. My concern is only that the baseline should match the real current execution path. Right now this benchmark uses a manual per-row fallback that creates a 1-row array for each row, while DataFusion’s generic grouped fallback batches rows per group. That means the follow-up PR may show gains against a heavier synthetic baseline. Could we either benchmark the real fallback path, or rename this to make clear it is only a synthetic baseline?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3077105775


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   Yes but there isnt s group accumulator and the point of the follow up pr is to add a group accumulator and these benches should hopefully show gains



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3077345404


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   I think the better approach is to benchmark the real grouped fallback adapter here, instead of the manual per-row loop. That is the path DataFusion uses today when COUNT(DISTINCT) has no native group accumulator, so it gives a fair before/after baseline for the follow-up PR. I would keep the current if supported { native } else { fallback adapter } shape. That way this benchmark measures the real current path now, and will automatically measure the optimized path once the follow-up lands.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3077100747


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   This still looks misleading to me. COUNT(DISTINCT) does not support `groups_accumulator_supported` yet, so this benchmark always runs the fallback path. The fallback creates a 1-row array for each input row, so it mostly measures small allocations and per-row updates, not real grouped distinct aggregation. Please either rename the benchmark to make that clear, or change the fallback so it matches the grouped case more closely.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3093533497


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,107 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    let group_counts = [100, 1000, 10000];
+    let cardinalities = [("low", 20), ("mid", 80), ("high", 99)];
+    let distributions = ["uniform", "skewed"];
+
+    for num_groups in group_counts {
+        for (card_name, distinct_pct) in cardinalities {
+            for dist in distributions {
+                let name = format!("g{num_groups}_{card_name}_{dist}");
+                let n_distinct = BATCH_SIZE * distinct_pct / 100;
+                let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+                let group_indices = if dist == "uniform" {
+                    create_uniform_groups(num_groups)
+                } else {
+                    create_skewed_groups(num_groups)
+                };
+
+                let (_schema, args) = prepare_args(DataType::Int64);
+
+                if count_fn.groups_accumulator_supported(args.clone()) {
+                    c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                        b.iter(|| {
+                            let mut acc =
+                                count_fn.create_groups_accumulator(args.clone()).unwrap();
+                            acc.update_batch(
+                                std::slice::from_ref(&values),
+                                &group_indices,
+                                None,
+                                num_groups,
+                            )
+                            .unwrap();
+                            acc.evaluate(EmitTo::All).unwrap()

Review Comment:
   ```suggestion
                               std::hint::black_box(acc.evaluate(EmitTo::All).unwrap())
   ```
   blackbox the result to protect from dead code elimination



##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,107 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)

Review Comment:
   ```suggestion
                   rng.random_range(hot_groups..num_groups)
   ```



##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,107 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    let group_counts = [100, 1000, 10000];
+    let cardinalities = [("low", 20), ("mid", 80), ("high", 99)];
+    let distributions = ["uniform", "skewed"];
+
+    for num_groups in group_counts {
+        for (card_name, distinct_pct) in cardinalities {
+            for dist in distributions {
+                let name = format!("g{num_groups}_{card_name}_{dist}");
+                let n_distinct = BATCH_SIZE * distinct_pct / 100;
+                let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+                let group_indices = if dist == "uniform" {
+                    create_uniform_groups(num_groups)
+                } else {
+                    create_skewed_groups(num_groups)
+                };
+
+                let (_schema, args) = prepare_args(DataType::Int64);
+
+                if count_fn.groups_accumulator_supported(args.clone()) {
+                    c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                        b.iter(|| {
+                            let mut acc =
+                                count_fn.create_groups_accumulator(args.clone()).unwrap();
+                            acc.update_batch(
+                                std::slice::from_ref(&values),
+                                &group_indices,
+                                None,
+                                num_groups,
+                            )
+                            .unwrap();
+                            acc.evaluate(EmitTo::All).unwrap()
+                        })
+                    });
+                } else {
+                    let arr = values.as_any().downcast_ref::<Int64Array>().unwrap();
+                    let mut group_rows: Vec<Vec<i64>> = vec![Vec::new(); num_groups];
+                    for (idx, &group_idx) in group_indices.iter().enumerate() {
+                        if arr.is_valid(idx) {
+                            group_rows[group_idx].push(arr.value(idx));
+                        }
+                    }
+                    let group_arrays: Vec<ArrayRef> = group_rows
+                        .iter()
+                        .map(|rows| Arc::new(Int64Array::from(rows.clone())) as ArrayRef)
+                        .collect();
+
+                    c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                        b.iter(|| {
+                            let mut accumulators: Vec<_> = (0..num_groups)
+                                .map(|_| prepare_accumulator(DataType::Int64))
+                                .collect();
+
+                            for (group_idx, batch) in group_arrays.iter().enumerate() {
+                                if !batch.is_empty() {
+                                    accumulators[group_idx]
+                                        .update_batch(std::slice::from_ref(batch))
+                                        .unwrap();
+                                }
+                            }
+
+                            let _results: Vec<_> = accumulators
+                                .iter_mut()
+                                .map(|acc| acc.evaluate().unwrap())
+                                .collect();

Review Comment:
   ```suggestion
                               let results: Vec<_> = accumulators
                                   .iter_mut()
                                   .map(|acc| acc.evaluate().unwrap())
                                   .collect();
                               std::hint::black_box(results);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3093663621


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,107 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    let group_counts = [100, 1000, 10000];
+    let cardinalities = [("low", 20), ("mid", 80), ("high", 99)];
+    let distributions = ["uniform", "skewed"];
+
+    for num_groups in group_counts {
+        for (card_name, distinct_pct) in cardinalities {
+            for dist in distributions {
+                let name = format!("g{num_groups}_{card_name}_{dist}");
+                let n_distinct = BATCH_SIZE * distinct_pct / 100;
+                let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+                let group_indices = if dist == "uniform" {
+                    create_uniform_groups(num_groups)
+                } else {
+                    create_skewed_groups(num_groups)
+                };
+
+                let (_schema, args) = prepare_args(DataType::Int64);

Review Comment:
   The usage of Box::leak() feels hacky.
   I'd just inline the body of prepare_args() here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3079916419


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {
+            c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                b.iter(|| {
+                    let mut acc =
+                        count_fn.create_groups_accumulator(args.clone()).unwrap();
+                    acc.update_batch(
+                        std::slice::from_ref(&values),
+                        &group_indices,
+                        None,
+                        num_groups,
+                    )
+                    .unwrap();
+                    acc.evaluate(EmitTo::All).unwrap()
+                })
+            });
+        } else {
+            c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                b.iter(|| {
+                    let mut accumulators: Vec<_> = (0..num_groups)
+                        .map(|_| prepare_accumulator(DataType::Int64))
+                        .collect();
+
+                    let arr = values.as_any().downcast_ref::<Int64Array>().unwrap();
+                    for (idx, group_idx) in group_indices.iter().enumerate() {
+                        if let Some(val) = arr.value(idx).into() {
+                            let single_val =
+                                Arc::new(Int64Array::from(vec![Some(val)])) as ArrayRef;
+                            accumulators[*group_idx]
+                                .update_batch(std::slice::from_ref(&single_val))
+                                .unwrap();
+                        }

Review Comment:
   ```suggestion
                           let single_val = values.slice(idx, 1);
                           accumulators[*group_idx]
                               .update_batch(std::slice::from_ref(&single_val))
                               .unwrap();
   ```
   slightly simpler and would avoid the allocation of the single valued array for each row



##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {
+            c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                b.iter(|| {
+                    let mut acc =
+                        count_fn.create_groups_accumulator(args.clone()).unwrap();
+                    acc.update_batch(
+                        std::slice::from_ref(&values),
+                        &group_indices,
+                        None,
+                        num_groups,
+                    )
+                    .unwrap();
+                    acc.evaluate(EmitTo::All).unwrap()
+                })
+            });
+        } else {
+            c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                b.iter(|| {
+                    let mut accumulators: Vec<_> = (0..num_groups)
+                        .map(|_| prepare_accumulator(DataType::Int64))
+                        .collect();
+
+                    let arr = values.as_any().downcast_ref::<Int64Array>().unwrap();
+                    for (idx, group_idx) in group_indices.iter().enumerate() {
+                        if let Some(val) = arr.value(idx).into() {
+                            let single_val =
+                                Arc::new(Int64Array::from(vec![Some(val)])) as ArrayRef;
+                            accumulators[*group_idx]
+                                .update_batch(std::slice::from_ref(&single_val))
+                                .unwrap();
+                        }

Review Comment:
   Another way would be to collect per-group indices first and then build the array:
   ```rust
   let mut group_rows: Vec<Vec<i64>> = vec![Vec::new(); num_groups];
   for (idx, &group_idx) in group_indices.iter().enumerate() {
       if arr.is_valid(idx) {
           group_rows[group_idx].push(arr.value(idx));
       }
   }
   for (group_idx, rows) in group_rows.iter().enumerate() {
       if !rows.is_empty() {
           let batch = Arc::new(Int64Array::from(rows.clone())) as ArrayRef;
           accumulators[group_idx].update_batch(std::slice::from_ref(&batch)).unwrap();
       }
   }
   ```



##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {
+            c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                b.iter(|| {
+                    let mut acc =
+                        count_fn.create_groups_accumulator(args.clone()).unwrap();
+                    acc.update_batch(
+                        std::slice::from_ref(&values),
+                        &group_indices,
+                        None,
+                        num_groups,
+                    )
+                    .unwrap();
+                    acc.evaluate(EmitTo::All).unwrap()
+                })
+            });
+        } else {
+            c.bench_function(&format!("count_distinct_groups {name}"), |b| {
+                b.iter(|| {
+                    let mut accumulators: Vec<_> = (0..num_groups)
+                        .map(|_| prepare_accumulator(DataType::Int64))
+                        .collect();
+
+                    let arr = values.as_any().downcast_ref::<Int64Array>().unwrap();
+                    for (idx, group_idx) in group_indices.iter().enumerate() {
+                        if let Some(val) = arr.value(idx).into() {

Review Comment:
   Why do you need an Option here ?
   `arr.value(idx)` returns i64 and calling `.into()` always return `Some`.
   If you want to filter out the nulls then you need to use `arr.is_null(idx)`



##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {

Review Comment:
   nit: Introduce an enum instead of using strings: 
   ```rust
   enum GroupDist {
       Uniform, 
       Skewed 
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3076813460


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   Thanks for the comment! I kept the name the same on purpose to be able to run benches before and after groups accumulator is implemented and get a direct comparison.  Keeping the name constant would show the before and after difference between the approaches while continuing to keep the same name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3071868071


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -87,6 +87,30 @@ fn create_i16_array(n_distinct: usize) -> Int16Array {
         .collect()
 }
 
+fn prepare_args(data_type: DataType) -> (Arc<Schema>, AccumulatorArgs<'static>) {

Review Comment:
   This helper uses Box::leak, and in the grouped path it is called inside b.iter. When COUNT(DISTINCT) gets real group accumulator support, every benchmark iteration will leak  memory. That will affect the benchmark result and make it less reliable. Please avoid leaking here, or build the args once outside the timed loop.



##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   This benchmark name is a bit misleading right now. COUNT(DISTINCT) still does not support groups_accumulator_supported, so this code always runs the fallback path. The fallback creates a new one-row array for every input row, so the benchmark mostly measures tiny allocations and per-row updates, not real grouped distinct aggregation. Please either rename the benchmark to make that clear, or change the fallback so it measures something closer to the real grouped case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3076802536


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -87,6 +87,30 @@ fn create_i16_array(n_distinct: usize) -> Int16Array {
         .collect()
 }
 
+fn prepare_args(data_type: DataType) -> (Arc<Schema>, AccumulatorArgs<'static>) {

Review Comment:
   done !  Thank you



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] chore: add count distinct group benchmarks [datafusion]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21575:
URL: https://github.com/apache/datafusion/pull/21575#discussion_r3077314158


##########
datafusion/functions-aggregate/benches/count_distinct.rs:
##########
@@ -150,5 +174,101 @@ fn count_distinct_benchmark(c: &mut Criterion) {
     });
 }
 
-criterion_group!(benches, count_distinct_benchmark);
+/// Create group indices with uniform distribution
+fn create_uniform_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    (0..BATCH_SIZE)
+        .map(|_| rng.random_range(0..num_groups))
+        .collect()
+}
+
+/// Create group indices with skewed distribution (80% in 20% of groups)
+fn create_skewed_groups(num_groups: usize) -> Vec<usize> {
+    let mut rng = StdRng::seed_from_u64(42);
+    let hot_groups = (num_groups / 5).max(1);
+    (0..BATCH_SIZE)
+        .map(|_| {
+            if rng.random_range(0..100) < 80 {
+                rng.random_range(0..hot_groups)
+            } else {
+                rng.random_range(0..num_groups)
+            }
+        })
+        .collect()
+}
+
+fn count_distinct_groups_benchmark(c: &mut Criterion) {
+    let count_fn = Count::new();
+
+    // bench different scenarios
+    let scenarios = [
+        // (name, num_groups, distinct_pct, group_fn)
+        ("sparse_uniform", 10, 80, "uniform"),
+        ("moderate_uniform", 100, 80, "uniform"),
+        ("dense_uniform", 1000, 80, "uniform"),
+        ("sparse_skewed", 10, 80, "skewed"),
+        ("dense_skewed", 1000, 80, "skewed"),
+        ("sparse_high_cardinality", 10, 99, "uniform"),
+        ("dense_low_cardinality", 1000, 20, "uniform"),
+    ];
+
+    for (name, num_groups, distinct_pct, group_type) in scenarios {
+        let n_distinct = BATCH_SIZE * distinct_pct / 100;
+        let values = Arc::new(create_i64_array(n_distinct)) as ArrayRef;
+        let group_indices = if group_type == "uniform" {
+            create_uniform_groups(num_groups)
+        } else {
+            create_skewed_groups(num_groups)
+        };
+
+        let (_schema, args) = prepare_args(DataType::Int64);
+
+        if count_fn.groups_accumulator_supported(args.clone()) {

Review Comment:
   Sure what do you think is the better approach here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org