You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "bharadwaj-pendyala (via GitHub)" <gi...@apache.org> on 2026/09/29 14:00:22 UTC

[PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

bharadwaj-pendyala opened a new pull request, #25876:
URL: https://github.com/apache/datafusion/pull/25876

   ## Which issue does this PR close?
   
   - Closes #25374.
   
   ## Rationale for this change
   
   `AggregateUDFImpl::distinct_handling` (#25288) doesn't survive the FFI boundary. `FFI_AggregateUDF` has no slot for it, so `ForeignAggregateUDF` falls back to the trait default and every UDAF loaded from another library reports `Sensitive`. A foreign `min` never gets its `DISTINCT` dropped, and a foreign `stddev`, which doesn't read `is_distinct` at all, tells the planner it deduplicates its own input.
   
   ## What changes are included in this PR?
   
   Same shape as `order_sensitivity`, all in `datafusion/ffi/src/udaf/mod.rs`:
   
   - `FFI_DistinctHandling`, a `#[repr(C)]` enum with `From` impls both ways, next to `FFI_AggregateOrderSensitivity`.
   - A `distinct_handling` fn pointer on `FFI_AggregateUDF`, added at the end of the struct after `supports_null_handling_clause`.
   - `ForeignAggregateUDF::distinct_handling` calls through it.
   
   `DistinctHandling` is `#[non_exhaustive]`, so the native-to-FFI conversion needs a wildcard arm. I map anything unknown to `Sensitive`, the trait default, which is what the issue suggested. That and appending the field at the end are the two assumptions worth a second look. The arm covers a variant added to `datafusion-expr` before the FFI enum catches up. Mismatched library versions stay unsupported, same as today.
   
   ## What is the testing strategy for this PR?
   
   - `udaf::tests::test_distinct_handling` wraps `min`, `sum` and `stddev` with the mock foreign marker and checks all three variants come back. On main `min` reports `Sensitive` and the test fails.
   - `tests/ffi_udaf.rs::test_distinct_handling` does the same through the separately built test library, for `stddev` (`Unsupported`) and `sum` (`Sensitive`). On main's `udaf/mod.rs` it fails with `left: Sensitive, right: Unsupported`.
   - `test_round_trip_all_distinct_handlings` round-trips each variant, like the existing order sensitivity test.
   
   `cargo test -p datafusion-ffi --features integration-tests` passes, as does `cargo clippy -p datafusion-ffi --all-targets --features integration-tests --no-deps -- -D warnings` and `cargo fmt --all -- --check`.
   
   ## Are there any user-facing changes?
   
   Foreign UDAFs now report the `distinct_handling` they declare, so an `Insensitive` one can have `DISTINCT` eliminated. Adding a field changes the `FFI_AggregateUDF` layout, so this probably wants the `api change` label.
   
   This was written with AI assistance. I've read the change end to end and reproduced both failures above myself.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#issuecomment-6310006895

   @bharadwaj-pendyala 
   Can you resolve the merge conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#issuecomment-5894019786

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25876?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `90.90909%` with `4 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.57%. Comparing base ([`0af6818`](https://app.codecov.io/gh/apache/datafusion/commit/0af68188912349e9c865548458b37b0e66b5c31d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e83de0b`](https://app.codecov.io/gh/apache/datafusion/commit/e83de0b77c5017816e54d5e083ac625bc77cb909?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 6 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25876?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/ffi/src/udaf/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25876?src=pr&el=tree&filepath=datafusion%2Fffi%2Fsrc%2Fudaf%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mZmkvc3JjL3VkYWYvbW9kLnJz) | 90.90% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25876?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #25876      +/-   ##
   ==========================================
   - Coverage   82.58%   82.57%   -0.01%     
   ==========================================
     Files        1142     1142              
     Lines      440930   440974      +44     
     Branches   440930   440974      +44     
   ==========================================
   + Hits       364123   364153      +30     
   - Misses      54804    54815      +11     
   - Partials    22003    22006       +3     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25876?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "bharadwaj-pendyala (via GitHub)" <gi...@apache.org>.
bharadwaj-pendyala commented on code in PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#discussion_r4325543938


##########
datafusion/ffi/tests/ffi_udaf.rs:
##########
@@ -82,6 +82,27 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn test_distinct_handling() -> Result<()> {
+        let module = get_module()?;
+
+        let ffi_stddev_func = (module.create_stddev_udaf)();
+        let foreign_stddev_func: Arc<dyn AggregateUDFImpl> = (&ffi_stddev_func).into();
+        assert_eq!(
+            foreign_stddev_func.distinct_handling(),
+            DistinctHandling::Unsupported
+        );
+

Review Comment:
   Added in d37014921. The test library now exposes `MIN` as `create_min_udaf`, and `test_foreign_min_distinct_drops_distinct` runs `MIN(DISTINCT b)` grouped by `a` through the foreign UDAF. It checks the result and that the optimized plan is a single `Aggregate` with `min(?table?.b)`, so no `DISTINCT` and no extra dedup aggregate.
   
   I also ran it with the forwarding change reverted. It fails there because the plan keeps `min(DISTINCT ?table?.b)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "bharadwaj-pendyala (via GitHub)" <gi...@apache.org>.
bharadwaj-pendyala commented on PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#issuecomment-6354053201

   @kosiew done, merged main in 2f3891e. #26564 moved `ffi/tests` into the new integration-tests crate, so the two tests now live in `integration-tests/tests/cases/udaf.rs`. `cargo xtask ci step test ffi` passes locally, 34 of 34 in that crate.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#discussion_r4324022667


##########
datafusion/ffi/tests/ffi_udaf.rs:
##########
@@ -82,6 +82,27 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn test_distinct_handling() -> Result<()> {
+        let module = get_module()?;
+
+        let ffi_stddev_func = (module.create_stddev_udaf)();
+        let foreign_stddev_func: Arc<dyn AggregateUDFImpl> = (&ffi_stddev_func).into();
+        assert_eq!(
+            foreign_stddev_func.distinct_handling(),
+            DistinctHandling::Unsupported
+        );
+

Review Comment:
   It would be useful to expose `MIN` through the test library and verify that foreign `MIN(DISTINCT x)` produces the correct result and an optimized plan without DISTINCT or an extra deduplication aggregate. This would cover the planner behavior across the actual library boundary, but the existing tests already verify the metadata forwarding.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#issuecomment-6359464573

   @timsaucer 
   Would like to take a look too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #25876:
URL: https://github.com/apache/datafusion/pull/25876


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(ffi): forward distinct_handling through FFI_AggregateUDF [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25876:
URL: https://github.com/apache/datafusion/pull/25876#issuecomment-5893578259

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-ffi v55.1.0 (current)
          Built [  56.762s] (current)
        Parsing datafusion-ffi v55.1.0 (current)
         Parsed [   0.064s] (current)
       Building datafusion-ffi v55.1.0 (baseline)
          Built [  56.513s] (baseline)
        Parsing datafusion-ffi v55.1.0 (baseline)
         Parsed [   0.063s] (baseline)
       Checking datafusion-ffi v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.274s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field FFI_AggregateUDF.distinct_handling in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/udaf/mod.rs:155
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [ 115.342s] datafusion-ffi
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org