You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "choplin (via GitHub)" <gi...@apache.org> on 2026/06/21 03:21:34 UTC

[PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

choplin opened a new pull request, #23063:
URL: https://github.com/apache/datafusion/pull/23063

   ## Which issue does this PR close?
   
   - Closes #14431
   
   ## Rationale for this change
   
   When a scalar or aggregate function is called with argument types that don't match any supported signature (e.g. `SELECT sum('a')`), the error message is a raw text dump with no source location. This PR attaches a `Diagnostic` so downstream consumers can highlight the offending call site in the original SQL query.
   
   ## What changes are included in this PR?
   
   - Add `spans: Spans` field to `ScalarFunction` and `AggregateFunction` so function-call source locations survive planning and optimization.
   - Populate spans from sqlparser in the SQL planner when `collect_spans` is enabled, following the same pattern used for `Column`.
   - In `verify_function_arguments`, attach a `Diagnostic` with:
     - **message**: the function name (`invalid argument type(s) for 'sum'`)
     - **span**: the function call site in the original SQL
     - **note**: the actual argument types (`called with argument type(s): Utf8`)
     - **help**: the candidate signatures (`candidate function(s): sum(Decimal), sum(UInt64), ...`)
   - Update all pattern matches, tree-node transforms, and serialization code for the new `spans` field.
   
   ## Are there any user-facing changes?
   
   No breaking API changes. `ScalarFunction` and `AggregateFunction` gain a `spans: Spans` field, but `Spans` is ignored in `PartialEq` / `Hash` and defaults to empty, so existing code using `ScalarFunction::new_udf` / `AggregateFunction::new_udf` is unaffected.
   
   When an error occurs, a diagnositc is attached in the error message.
   
   **Before** (no `Diagnostic` attached):
   ```text
   Error during planning: No function matches the given name and argument types 'sum(Utf8)'. You might need to add explicit type casts.
   	Candidate functions:
   	sum(Decimal)
   	sum(UInt64)
   	sum(Int64)
   	sum(Float64)
   	sum(Duration)
   ```
   
   **After** (same error message, with `Diagnostic` now attached):
   ```text
   Error during planning: No function matches the given name and argument types 'sum(Utf8)'. You might need to add explicit type casts.
   	Candidate functions:
   	sum(Decimal)
   	sum(UInt64)
   	sum(Int64)
   	sum(Float64)
   	sum(Duration)
   
   Diagnostic:
     message: invalid argument type(s) for 'sum'
     span:    1:8 - 1:11
     note:    called with argument type(s): Utf8
     help:    candidate function(s): sum(Decimal), sum(UInt64), sum(Int64), sum(Float64), sum(Duration)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3636937070


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Between the two options you mentioned, I'd lean toward keeping the parser-only span information out of these public struct layouts. Validating in SqlToRel, where the call-site span is already available, seems like a good fit and avoids introducing a semver break while still preserving the improved diagnostics. If there's another semver-compatible way to associate the span with these expressions, that would also work, but I don't think adding the new public fields is something we should ship in a patch release.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3563660235


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Could we avoid adding `spans` as a required public field on these expression structs?
   
   `ScalarFunction`, `AggregateFunction`, and `WindowFunction` are publicly constructible with struct literals. Adding a new required field means existing downstream code using those literals will no longer compile. `cargo-semver-checks` reports `constructible_struct_adds_field` for `ScalarFunction.spans`, `AggregateFunction.spans`, and `WindowFunction.spans`.
   
   Please keep the diagnostic span data outside these public struct layouts, or use another semver-compatible design. The new required public fields should be removed before this is merged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#issuecomment-4842550245

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion v54.0.0 (current)
          Built [  98.163s] (current)
        Parsing datafusion v54.0.0 (current)
         Parsed [   0.034s] (current)
       Building datafusion v54.0.0 (baseline)
          Built [  97.886s] (baseline)
        Parsing datafusion v54.0.0 (baseline)
         Parsed [   0.035s] (baseline)
       Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.567s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [ 198.373s] datafusion
       Building datafusion-expr v54.0.0 (current)
          Built [  25.494s] (current)
        Parsing datafusion-expr v54.0.0 (current)
         Parsed [   0.073s] (current)
       Building datafusion-expr v54.0.0 (baseline)
          Built [  25.424s] (baseline)
        Parsing datafusion-expr v54.0.0 (baseline)
         Parsed [   0.074s] (baseline)
       Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   1.161s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
   
   --- failure constructible_struct_adds_field: externally-constructible struct adds field ---
   
   Description:
   A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ScalarFunction.spans in /home/runner/work/datafusion/datafusion/datafusion/expr/src/expr.rs:934
     field AggregateFunction.spans in /home/runner/work/datafusion/datafusion/datafusion/expr/src/expr.rs:1109
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  54.558s] datafusion-expr
       Building datafusion-functions v54.0.0 (current)
          Built [  29.154s] (current)
        Parsing datafusion-functions v54.0.0 (current)
         Parsed [   0.080s] (current)
       Building datafusion-functions v54.0.0 (baseline)
          Built [  28.746s] (baseline)
        Parsing datafusion-functions v54.0.0 (baseline)
         Parsed [   0.082s] (baseline)
       Checking datafusion-functions v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.417s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  59.530s] datafusion-functions
       Building datafusion-functions-aggregate v54.0.0 (current)
          Built [  29.621s] (current)
        Parsing datafusion-functions-aggregate v54.0.0 (current)
         Parsed [   0.042s] (current)
       Building datafusion-functions-aggregate v54.0.0 (baseline)
          Built [  29.321s] (baseline)
        Parsing datafusion-functions-aggregate v54.0.0 (baseline)
         Parsed [   0.045s] (baseline)
       Checking datafusion-functions-aggregate v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.170s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  60.435s] datafusion-functions-aggregate
       Building datafusion-functions-nested v54.0.0 (current)
          Built [  34.225s] (current)
        Parsing datafusion-functions-nested v54.0.0 (current)
         Parsed [   0.035s] (current)
       Building datafusion-functions-nested v54.0.0 (baseline)
          Built [  34.906s] (baseline)
        Parsing datafusion-functions-nested v54.0.0 (baseline)
         Parsed [   0.039s] (baseline)
       Checking datafusion-functions-nested v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.204s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  70.328s] datafusion-functions-nested
       Building datafusion-optimizer v54.0.0 (current)
          Built [  25.646s] (current)
        Parsing datafusion-optimizer v54.0.0 (current)
         Parsed [   0.028s] (current)
       Building datafusion-optimizer v54.0.0 (baseline)
          Built [  25.651s] (baseline)
        Parsing datafusion-optimizer v54.0.0 (baseline)
         Parsed [   0.031s] (baseline)
       Checking datafusion-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.162s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  52.945s] datafusion-optimizer
       Building datafusion-physical-expr v54.0.0 (current)
          Built [  27.374s] (current)
        Parsing datafusion-physical-expr v54.0.0 (current)
         Parsed [   0.045s] (current)
       Building datafusion-physical-expr v54.0.0 (baseline)
          Built [  27.322s] (baseline)
        Parsing datafusion-physical-expr v54.0.0 (baseline)
         Parsed [   0.047s] (baseline)
       Checking datafusion-physical-expr v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.349s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  56.239s] datafusion-physical-expr
       Building datafusion-proto v54.0.0 (current)
          Built [  56.353s] (current)
        Parsing datafusion-proto v54.0.0 (current)
         Parsed [   0.017s] (current)
       Building datafusion-proto v54.0.0 (baseline)
          Built [  56.344s] (baseline)
        Parsing datafusion-proto v54.0.0 (baseline)
         Parsed [   0.017s] (baseline)
       Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.238s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [ 114.009s] datafusion-proto
       Building datafusion-sql v54.0.0 (current)
          Built [  39.429s] (current)
        Parsing datafusion-sql v54.0.0 (current)
         Parsed [   0.029s] (current)
       Building datafusion-sql v54.0.0 (baseline)
          Built [  39.892s] (baseline)
        Parsing datafusion-sql v54.0.0 (baseline)
         Parsed [   0.031s] (baseline)
       Checking datafusion-sql v54.0.0 -> v54.0.0 (no change; assume patch)
        Checked [   0.237s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [  80.559s] datafusion-sql
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3563656975


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Could we avoid adding `spans` as a required public field on these expression structs?
   
   `ScalarFunction`, `AggregateFunction`, and `WindowFunction` are publicly constructible with struct literals. Adding a new required field means existing downstream code using those literals will no longer compile. `cargo-semver-checks` reports `constructible_struct_adds_field` for `ScalarFunction.spans`, `AggregateFunction.spans`, and `WindowFunction.spans`.
   
   Please keep the diagnostic span data outside these public struct layouts, or use another semver-compatible design. The new required public fields should be removed before this is merged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3563660235


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Could we avoid adding `spans` as a required public field on these expression structs?
   
   `ScalarFunction`, `AggregateFunction`, and `WindowFunction` are publicly constructible with struct literals. Adding a new required field means existing downstream code using those literals will no longer compile. `cargo-semver-checks` reports `constructible_struct_adds_field` for `ScalarFunction.spans`, `AggregateFunction.spans`, and `WindowFunction.spans`.
   
   Please keep the diagnostic span data outside these public struct layouts, or use another semver-compatible design. The new required public fields should be removed before this is merged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3755147586


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   I would advise:
   
   1. **Separate the work.** Keep this PR patch-compatible: no `spans` fields on `ScalarFunction`, `AggregateFunction`, or `WindowFunction`. The major-version PR should contain only the API/layout change and the propagation required for it, rather than bundling a diagnostic behavior change with a release break.
   
   2. **Coordinate the release boundary first.** Ask maintainers/release owners to confirm the target major release and its versioning plan. The version in the relevant public crate manifest(s) must be advanced as part of that coordinated major-release work. [Version 55 is in progress already](https://github.com/apache/datafusion/issues/22393), so this may have to wait till version 56.
   
   3. **Make the break explicit.** State in the PR description and release notes that public struct literals are source-incompatible. Show the downstream migration:
   
      ```rust
      // Before
      ScalarFunction { func, args }
   
      // After (if a literal is still required)
      ScalarFunction {
          func,
          args,
          spans: Spans::new(),
      }
   
      // Prefer the stable constructor where it expresses the use case
      ScalarFunction::new_udf(func, args)
      ```
   
      Equivalent literal users of `AggregateFunction` and `WindowFunction` need the same update. Add or retain constructors that initialize empty spans so normal programmatic expression construction does not need parser metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#issuecomment-4829981464

   @choplin 
   can you resolve the merge conflict?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3550126661


##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -549,13 +549,13 @@ impl ExprSchemable for Expr {
                 match fun {
                     WindowFunctionDefinition::AggregateUDF(udaf) => {
                         let new_fields =
-                            verify_function_arguments(udaf.as_ref(), &fields)?;
+                            verify_function_arguments(udaf.as_ref(), &fields, None)?;

Review Comment:
   Thanks! Fixed in c8a44313a.
   
   WindowFunction now carries a spans field (mirroring `ScalarFunction` / `AggregateFunction`). `SqlToRel` populates it from the function-call token when `collect_spans` is enabled, and both the `AggregateUDF` and `WindowUDF` window branches now pass `spans.first()` into `verify_function_arguments` instead of None, so `sum(first_name) OVER ()` and window-UDF calls point at the SQL call site.
   
   Added `test_invalid_window_function_argument_types` based on your example `(sum(first_name) OVER ())`, asserting the diagnostic points at the sum call site.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3563665326


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Could we avoid adding `spans` as a required public field on these expression structs?
   
   `ScalarFunction`, `AggregateFunction`, and `WindowFunction` are public and can be constructed with struct literals. Adding a new required field means existing downstream code using those literals will no longer compile. The semver check reports `constructible_struct_adds_field` for all three structs.
   
   Please keep the diagnostic span storage outside these public constructible layouts, or use another semver-compatible design. The new required public fields should be removed before merge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#issuecomment-4833765061

   @kosiew I rebased onto the latest main HEAD and resolved the conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3563665326


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Could we avoid adding `spans` as a required public field on these expression structs?
   
   `ScalarFunction`, `AggregateFunction`, and `WindowFunction` are public and can be constructed with struct literals. Adding a new required field means existing downstream code using those literals will no longer compile. The semver check reports `constructible_struct_adds_field` for all three structs.
   
   Please keep the diagnostic span storage outside these public constructible layouts, or use another semver-compatible design. The new required public fields should be removed before merge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3755147586


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   @choplin 
   
   I would advise:
   
   1. **Separate the work.** Keep this PR patch-compatible: no `spans` fields on `ScalarFunction`, `AggregateFunction`, or `WindowFunction`. The major-version PR should contain only the API/layout change and the propagation required for it, rather than bundling a diagnostic behavior change with a release break.
   
   2. **Coordinate the release boundary first.** Ask maintainers/release owners to confirm the target major release and its versioning plan. The version in the relevant public crate manifest(s) must be advanced as part of that coordinated major-release work. [Version 55 is in progress already](https://github.com/apache/datafusion/issues/22393), so this may have to wait till version 56.
   
   3. **Make the break explicit.** State in the PR description and release notes that public struct literals are source-incompatible. Show the downstream migration:
   
      ```rust
      // Before
      ScalarFunction { func, args }
   
      // After (if a literal is still required)
      ScalarFunction {
          func,
          args,
          spans: Spans::new(),
      }
   
      // Prefer the stable constructor where it expresses the use case
      ScalarFunction::new_udf(func, args)
      ```
   
      Equivalent literal users of `AggregateFunction` and `WindowFunction` need the same update. Add or retain constructors that initialize empty spans so normal programmatic expression construction does not need parser metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3623581941


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Sorry for the slow reply here1
   
   You're right about the semver break — adding `spans` to these public structs trips
   `constructible_struct_adds_field`.
   
   My thinking so far: I noticed `Column` already carries the same `pub spans: Spans`
   field, added the same way in #13664 (the parent Diagnostics EPIC this PR belongs to),
   and it's likewise publicly constructible — so one direction would be to stay consistent
   with that precedent. Another would be to keep the span off these struct layouts entirely
   and validate arguments eagerly in `SqlToRel`, where the call-site span is already in
   scope, so the diagnostic is attached at planning time.
   
   I'm not sure which fits best here, though, and there may well be a cleaner
   semver-compatible approach I'm not seeing. Do you have a direction you'd prefer — or a
   better idea than either of these?



##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Sorry for the slow reply here!
   
   You're right about the semver break — adding `spans` to these public structs trips
   `constructible_struct_adds_field`.
   
   My thinking so far: I noticed `Column` already carries the same `pub spans: Spans`
   field, added the same way in #13664 (the parent Diagnostics EPIC this PR belongs to),
   and it's likewise publicly constructible — so one direction would be to stay consistent
   with that precedent. Another would be to keep the span off these struct layouts entirely
   and validate arguments eagerly in `SqlToRel`, where the call-site span is already in
   scope, so the diagnostic is attached at planning time.
   
   I'm not sure which fits best here, though, and there may well be a cleaner
   semver-compatible approach I'm not seeing. Do you have a direction you'd prefer — or a
   better idea than either of these?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3755227859


##########
datafusion/sql/src/expr/function.rs:
##########
@@ -362,10 +382,19 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
                     .map(|arg| arg.to_string())
                     .collect::<Vec<_>>()
                     .join(",");
-                let verbose_alias = format!("{name}({arg_names})");
-
-                return Ok(Expr::ScalarFunction(inner).alias(verbose_alias));
-            }
+                Some(format!("{name}({arg_names})"))
+            };
+            let expr = Expr::ScalarFunction(inner);
+            let span = if self.options.collect_spans {
+                Span::try_from_sqlparser_span(sql_parser_span)
+            } else {
+                None
+            };
+            validate_function_expr(&expr, schema, &name, span)?;

Review Comment:
   I think this validation is happening too early. `validate_function_expr` calls `Expr::to_field` while we are still in `SqlToRel`, before the `TypeCoercion` analyzer has inserted any necessary casts.
   
   That changes behavior for valid coercible calls. I reproduced it with the existing `select_where_nullif_division` test: `nullif(c4 + c5, 0)` reaches this point as `nullif(Int32, Int64)` and is rejected, even though the analyzer normally coerces the literal and the query should plan successfully.
   
   Could we avoid validating here and keep the existing post-coercion validation boundary? I would prefer attaching the best available argument or subexpression span when `verify_function_arguments` emits the diagnostic, even if that span is approximate, rather than changing when validation happens.



##########
datafusion/sql/src/expr/function.rs:
##########
@@ -81,6 +81,27 @@ fn find_closest_match(candidates: Vec<String>, target: &str) -> Option<String> {
     })
 }
 
+fn validate_function_expr(
+    expr: &Expr,
+    schema: &DFSchema,
+    function_name: &str,
+    function_span: Option<Span>,
+) -> Result<()> {
+    expr.to_field(schema).map(|_| ()).map_err(|err| {
+        let expected_message = format!("invalid argument type(s) for '{function_name}'");

Review Comment:
   I think aliases will miss the call-site span here. `function_name` is the spelling from the SQL query, but `verify_function_arguments` builds the diagnostic using the resolved UDF's canonical name.
   
   For example, `pow(first_name, first_name)` resolves to `power`, so the diagnostic says `invalid argument type(s) for 'power'`, while this helper looks for `invalid argument type(s) for 'pow'`. The messages do not match, so the original diagnostic is returned without the span.
   
   If this matching approach remains, could we compare against the canonical resolved function name while still attaching the span from the SQL call? It would also be good to add alias regression coverage for the applicable scalar, aggregate, and window paths.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#issuecomment-4923207688

   @kosiew Thanks for the review! I've addressed the two points from your review. Please take a look!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #23063: feat: attach Diagnostic to "invalid function argument types" error
URL: https://github.com/apache/datafusion/pull/23063


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3733346997


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   @kosiew Sorry, I've noticed that fix I reported in 879faea doesn't work.
   
   Validating in `SqlToRel` happens before the `TypeCoercion` analyzer rule, so it only sees the argument types as written. This rejects valid queries. The existing test `select_where_nullif_division` fails:
   
   ```sql
   SELECT c3/(c4+c5) FROM aggregate_test_100 WHERE c3/nullif(c4+c5, 0) > 0.1
   ```
   
   ```
   invalid argument type(s) for 'nullif'
     called with argument type(s): Int32, Int64
     candidate function(s): nullif(Int32, Int32)
   ```
   
   `nullif(Int32, Int64)` is coercible to `nullif(Int32, Int32)`, but the check runs too early to see the coerced types.
   
   The difficulty is that the error is only known where coercion runs (`verify_function_arguments`, during analysis), while the span is only known during planning. Something has to bring the two together. These are the four options I can think of:
   
   1. Make the check in `SqlToRel` coercion-aware, so it accepts arguments that coercion would accept. This keeps the current design and needs no struct change, but it reimplements what `TypeCoercion` already does, in the planner, just to produce a diagnostic. The two would have to stay in sync, and if they drift the planner rejects queries the engine can run.
   2. Point the diagnostic at the arguments. Arguments are `Expr::Column`s and already carry spans, so `verify_function_arguments` can take the span from them instead of from the call. Validation stays where it is, so valid queries are no longer rejected. The location becomes approximate: `sum(first_name)` is reported at `first_name`, and for a call like `sum(a + b)` it is reported at whichever column comes first inside the argument.
   3. Attach the span after analysis. The diagnostic is produced without a span, and a later step matches it against the parsed statement by function name to fill in the location. This changes no struct, but the match is a heuristic. For `SELECT sum(a), sum(b)` there are two calls to the same function and no reliable way to tell which one failed.
   4. Add `spans` to the expression structs, as the original approach did, and land it in a future major release.
   
   Option 4 is the one I would prefer. If the expression itself holds the span, the span is still available when the error is raised, so the check can stay in `verify_function_arguments` and run after coercion. The diagnostic then points at the call itself. It also follows `Column`, which has carried `pub spans: Spans` since #13664.
   
   If there is a better way to carry the span into analysis that I have missed, I would be glad to hear it.
   
   Your concern was about shipping the new public fields in a patch release. I read the API health policy, but I could not tell whether a contribution that has to wait for a major version bump is acceptable, or how such a change is usually handled here. Could you advise? Depending on that, I can hold this PR until the next major release, or land the diagnostic without a span now and add the span later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3623581941


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Sorry for the slow reply here!
   
   You're right about the semver break — adding `spans` to these public structs trips
   `constructible_struct_adds_field`.
   
   My thinking so far: I noticed `Column` already carries the same `pub spans: Spans`
   field, added the same way to that (publicly constructible) struct in the earlier
   diagnostics work (#13664) — so one direction would be to stay consistent with that
   precedent. Another would be to keep the span off these struct layouts entirely and
   validate arguments eagerly in `SqlToRel`, where the call-site span is already in scope,
   so the diagnostic is attached at planning time.
   
   I'm not sure which fits best here, though, and there may well be a cleaner
   semver-compatible approach I'm not seeing. Do you have a direction you'd prefer — or a
   better idea than either of these?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3725540789


##########
datafusion/expr/src/expr.rs:
##########
@@ -930,21 +930,32 @@ pub struct ScalarFunction {
     pub func: Arc<crate::ScalarUDF>,
     /// List of expressions to feed to the functions as arguments
     pub args: Vec<Expr>,
+    /// Original source code location, if known
+    pub spans: Spans,

Review Comment:
   Sorry this took so long. I removed the parser-only spans from the public expression structs and now validate scalar, aggregate, and window functions in SqlToRel, attaching the call-site span only to matching diagnostics. Fixed in 879faea.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "choplin (via GitHub)" <gi...@apache.org>.
choplin commented on code in PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#discussion_r3550130529


##########
datafusion/sql/tests/cases/diagnostic.rs:
##########
@@ -671,3 +693,16 @@ fn test_multiple_null_comparison_warnings() -> Result<()> {
     );
     Ok(())
 }
+
+#[test]

Review Comment:
   Done in 371b705d4 — added `test_invalid_scalar_function_argument_types`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: attach Diagnostic to "invalid function argument types" error [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23063:
URL: https://github.com/apache/datafusion/pull/23063#issuecomment-6104394392

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org