You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "akashjainn (via GitHub)" <gi...@apache.org> on 2026/09/03 20:15:08 UTC
[PR] fix: cap the field list in "No field named" errors [datafusion]
akashjainn opened a new pull request, #24931:
URL: https://github.com/apache/datafusion/pull/24931
Closes #5332.
## Which issue does this PR close?
Closes #5332.
## Rationale for this change
Selecting an unknown column from a wide table lists every field in the schema. The issue was filed in 2023 and still reproduces on `main` — on a 200 column table the message is ~2.8k characters:
```console
> CREATE TABLE wide(col_0 INT, ... col_199 INT) AS VALUES (...);
> SELECT not_a_column FROM wide;
Error: Schema error: No field named not_a_column.
Valid fields are wide.col_0, wide.col_1, ... wide.col_199. <- 200 names
```
Schemas with hundreds or thousands of columns are ordinary in analytics workloads, so the useful part of the message — the name that was not found, and the `Did you mean` suggestion — gets pushed off screen by the list that follows it.
## What changes are included in this PR?
Cap the listed fields at 20 and summarise the remainder. Same query after:
```console
Error: Schema error: No field named not_a_column.
Valid fields are wide.col_0, ... wide.col_19 and 180 others.
```
2.8k characters down to 356.
Schemas at or below the cap are unchanged, so the list stays useful where it is short enough to read:
```console
> SELECT alpah FROM t; -- t(alpha, beta, gamma)
Error: Schema error: No field named alpah. Did you mean 't.alpha'?
Valid fields are t.alpha, t.beta, t.gamma.
```
That also means the existing message assertions in `dfschema.rs`, `column.rs`, `expr_rewriter/mod.rs` and `core/tests/dataframe/mod.rs` are untouched — all of those schemas are well under 20 fields.
The issue suggested either dropping the list (as Postgres does) or truncating it. I kept it and bounded it: the list is genuinely helpful on narrow schemas, and `closest_valid_field` already covers the common typo case, so removing it outright would lose more than it gains. Happy to switch to the Postgres behaviour if maintainers prefer.
## Are these changes tested?
Yes — four unit tests in `datafusion/common/src/error.rs`:
| Test | Covers |
|---|---|
| `field_not_found_lists_every_field_for_a_narrow_schema` | 3 fields — full list, no "other" suffix |
| `field_not_found_lists_every_field_at_the_cap` | exactly 20 — boundary, still a full list |
| `field_not_found_uses_singular_for_one_extra_field` | 21 — reads "and 1 other." not "1 others" |
| `field_not_found_truncates_a_wide_schema` | 200 — "and 180 others.", asserts the message stays under 500 chars |
`cargo test -p datafusion-common` — **608 passed, 0 failed**. `cargo test -p datafusion-expr --lib` — 258 passed, 0 failed. `cargo fmt` clean.
Verified end to end with `datafusion-cli` built from this branch, output shown above.
Two notes on the local run: the `arrow_test_data` doctest fails in my checkout because the `testing/` submodule isn't initialised, and `cargo clippy -p datafusion-common --all-targets` reports a pre-existing `unused import: crate::config::TableParquetOptions`. Both reproduce on unmodified `main`, so neither is from this change.
## Are there any user-facing changes?
Yes — the wording of `SchemaError::FieldNotFound` when a schema has more than 20 fields. It gains a trailing `and N others.` instead of the full list. No API change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: cap the field list in "No field named" errors [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #24931:
URL: https://github.com/apache/datafusion/pull/24931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: cap the field list in "No field named" errors [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24931:
URL: https://github.com/apache/datafusion/pull/24931#issuecomment-5556822054
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24931?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `97.77778%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.64%. Comparing base ([`20d1c56`](https://app.codecov.io/gh/apache/datafusion/commit/20d1c5676118e0bca955ccf92c0d2cf665384d0d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`31c5184`](https://app.codecov.io/gh/apache/datafusion/commit/31c5184d0c4fe86377b5229b8e2d28603870c7f5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 21 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24931?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/common/src/error.rs](https://app.codecov.io/gh/apache/datafusion/pull/24931?src=pr&el=tree&filepath=datafusion%2Fcommon%2Fsrc%2Ferror.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL2Vycm9yLnJz) | 97.77% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24931?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24931 +/- ##
==========================================
- Coverage 81.64% 81.64% -0.01%
==========================================
Files 1123 1123
Lines 410248 410286 +38
Branches 410248 410286 +38
==========================================
+ Hits 334940 334971 +31
- Misses 55617 55622 +5
- Partials 19691 19693 +2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24931?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: cap the field list in "No field named" errors [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24931:
URL: https://github.com/apache/datafusion/pull/24931#issuecomment-5641017834
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-common v55.0.0 (current)
Built [ 29.986s] (current)
Parsing datafusion-common v55.0.0 (current)
Parsed [ 0.059s] (current)
Building datafusion-common v55.0.0 (baseline)
Built [ 29.270s] (baseline)
Parsing datafusion-common v55.0.0 (baseline)
Parsed [ 0.057s] (baseline)
Checking datafusion-common v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.876s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_missing: pub enum removed or renamed ---
Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_missing.ron
Failed in:
enum datafusion_common::parquet_config::DFParquetStatistics, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/9aa108929b2879c28f04d56b5bbe8b37c52f4a4a/datafusion/common/src/parquet_config.rs:115
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 61.636s] datafusion-common
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: cap the field list in "No field named" errors [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #24931:
URL: https://github.com/apache/datafusion/pull/24931#discussion_r3942865871
##########
datafusion/common/src/error.rs:
##########
@@ -1195,6 +1210,59 @@ pub fn add_possible_columns_to_diag(
#[cfg(test)]
mod test {
+ use crate::error::{Column, MAX_LISTED_VALID_FIELDS, SchemaError};
+
+ fn field_not_found_message(num_fields: usize) -> String {
+ let valid_fields = (0..num_fields)
+ .map(|i| Column::new_unqualified(format!("col_{i}")))
+ .collect::<Vec<_>>();
+ SchemaError::FieldNotFound {
+ field: Box::new(Column::new_unqualified("nope")),
+ valid_fields,
+ }
+ .to_string()
+ }
+
+ #[test]
Review Comment:
The tests mostly use contains, negative checks, and a length ceiling. They would still pass if fields 1–18 disappeared, punctuation broke, or unrelated text was appended.
How about using an insta snapshot for the complete 3-, 20-, 21-, and 200-field messages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: cap the field list in "No field named" errors [datafusion]
Posted by "akashjainn (via GitHub)" <gi...@apache.org>.
akashjainn commented on code in PR #24931:
URL: https://github.com/apache/datafusion/pull/24931#discussion_r3945483413
##########
datafusion/common/src/error.rs:
##########
@@ -1195,6 +1210,59 @@ pub fn add_possible_columns_to_diag(
#[cfg(test)]
mod test {
+ use crate::error::{Column, MAX_LISTED_VALID_FIELDS, SchemaError};
+
+ fn field_not_found_message(num_fields: usize) -> String {
+ let valid_fields = (0..num_fields)
+ .map(|i| Column::new_unqualified(format!("col_{i}")))
+ .collect::<Vec<_>>();
+ SchemaError::FieldNotFound {
+ field: Box::new(Column::new_unqualified("nope")),
+ valid_fields,
+ }
+ .to_string()
+ }
+
+ #[test]
Review Comment:
Thanks for pointing that out! fixed here: 92c606c2729a4cd2e7b507984e10117cdd359dbf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: cap the field list in "No field named" errors [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24931:
URL: https://github.com/apache/datafusion/pull/24931#issuecomment-5655911939
I don't understand the semver check reported failure: https://github.com/apache/datafusion/pull/24931#issuecomment-5641017834
It says
```
--- failure enum_missing: pub enum removed or renamed ---
Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_missing.ron
Failed in:
enum datafusion_common::parquet_config::DFParquetStatistics, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/9aa108929b2879c28f04d56b5bbe8b37c52f4a4a/datafusion/common/src/parquet_config.rs:115
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 61.636s] datafusion-common
```
But there are no changes to `DFParquetStatistics` 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org