You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/10/09 05:36:36 UTC
[PR] fix: match functional dependencies by column position, not by name [datafusion]
jayzhan211 opened a new pull request, #26150:
URL: https://github.com/apache/datafusion/pull/26150
## Which issue does this PR close?
- None yet. The bug and its repro are below.
## Rationale for this change
If `x` is a primary key, the optimizer treats `CAST(x AS ...)` as `x`. It then drops the other ORDER BY or GROUP BY keys that `x` determines, so the query returns wrong results:
```sql
CREATE TABLE t (x DOUBLE, y VARCHAR, PRIMARY KEY (x))
AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
SELECT x, y FROM t ORDER BY CAST(x AS INT), y;
-- main: 1.1 b / 1.2 a / 2.1 c (`y` is dropped from the sort)
-- expected: 1.2 a / 1.1 b / 2.1 c
SELECT CAST(x AS INT) k, count(*) n FROM t GROUP BY CAST(x AS INT), y;
-- main: (1, 2) / (2, 1) (`y` is dropped from the grouping)
-- expected: (1, 1) / (1, 1) / (2, 1)
SELECT y, count(*) FROM t GROUP BY CAST(x AS INT);
-- main: accepted
-- expected: Error: Column in SELECT must be in GROUP BY or an aggregate function
```
`CAST(x AS INT)` is named `t.x`, because casts are left out of expression names. The functional dependency helpers matched GROUP BY and ORDER BY expressions to key columns by comparing names, so the cast matched the key `t.x`.
The same happens with `TRY_CAST`, with a `UNIQUE NOT NULL` key, and with a key that comes from an inner GROUP BY. The ORDER BY case comes from the sort key pruning added in #21362 (54.0.0).
## What changes are included in this PR?
- **Index-based helpers.** The four helpers in `functional_dependencies.rs` now take, for each GROUP BY or ORDER BY expression, the index of the input field it references (`Option<usize>`) instead of its name. A computed expression has no index, so it never matches a key. The new `datafusion_expr::utils::passthrough_field_index` returns the index for a column reference, aliased or not.
- **One GROUP BY list.** The dependencies of an `Aggregate`'s output use the GROUP BY list that its schema is built from (`grouping_set_to_exprlist`). Before, they used a list de-duplicated by name, which merged `CAST(x AS INT)` and `x`.
- **Projections resolve columns.** A projection now finds each input column with `DFSchema::index_of_column_by_name` instead of comparing `"qualifier.name"` strings. The unit test `projection_duplicate_flattened_name_uses_first_input_index` asserted the old string behaviour: a column named `"orders.id"` got the dependency of the different field `orders.id`. It is renamed and now asserts that each column keeps its own dependency.
## What is the testing strategy for this PR?
- **New tests.** Section 6 of `functional_dependencies.slt` has one query for each user of the helpers:
- ORDER BY pruning;
- GROUP BY pruning;
- `Aggregate` output dependencies;
- GROUP BY expansion.
On `main`, each of these returns a wrong result or accepts an invalid query.
- **Existing tests** are unchanged, apart from the rewritten unit test.
- **Planning time.** Planning is not slower. The `sql_planner` TPC-H and TPC-DS benchmarks, whose tables have primary keys, are about 5% faster, because the new code no longer renders expression names.
<details><summary>Benchmark numbers (3 interleaved runs per side)</summary>
| Benchmark | `main` | This PR |
|---|---|---|
| `physical_plan_tpch_all` | 17.3 / 17.8 / 18.0 ms | 16.1 / 17.0 / 17.2 ms |
| `physical_plan_tpcds_all` | 291 / 300 / 301 ms | 276 / 280 / 282 ms |
</details>
## Are there any user-facing changes?
- **Results.** The queries above return correct results.
- **API change.** Four `pub` functions in `datafusion_common` take `&[Option<usize>]` instead of `&[String]`:
- `aggregate_functional_dependencies`
- `get_target_functional_dependencies`
- `get_required_group_by_exprs_indices`
- `get_required_sort_exprs_indices`
The upgrade guide has a migration note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4273918944
##########
datafusion/common/src/functional_dependencies.rs:
##########
@@ -480,24 +485,21 @@ pub fn aggregate_functional_dependencies(
{
// Indices into the GROUP BY list for this determinant:
let mut new_source_indices = vec![];
- let mut new_source_field_names = vec![];
- let source_field_names = source_indices
- .iter()
- .map(|&idx| &aggr_input_fields[idx])
- .collect::<Vec<_>>();
-
- for (idx, group_by_expr_name) in group_by_expr_names.iter().enumerate() {
- // When one of the input determinant expressions matches with
- // the GROUP BY expression, add the index of the GROUP BY
- // expression as a new determinant key:
- if source_field_names.contains(&group_by_expr_name) {
+ let mut new_source_input_indices = vec![];
+ for (idx, input_idx) in group_by_input_indices.iter().enumerate() {
+ // When one of the input determinant columns is a GROUP BY
+ // expression, add the index of the GROUP BY expression as a new
+ // determinant key:
+ if let Some(input_idx) = input_idx
+ && source_indices.contains(input_idx)
+ {
Review Comment:
addressed
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
+
+statement ok
+drop table t_wide;
+
+# 6.6 The same with `TRY_CAST` over a `UNIQUE NOT NULL` key: 'bad-a' and
+# 'bad-b' both cast to NULL.
+statement ok
+CREATE TABLE t_try_cast (x VARCHAR NOT NULL UNIQUE, y VARCHAR) AS VALUES ('bad-a', 'b'), ('bad-b', 'a'), ('1', 'c');
+
+query TT
+SELECT x, y FROM t_try_cast ORDER BY TRY_CAST(x AS INT), y;
+----
+1 c
+bad-b a
+bad-a b
+
+query IT rowsort
+SELECT TRY_CAST(x AS INT) k, y FROM t_try_cast GROUP BY TRY_CAST(x AS INT), y;
+----
+1 c
+NULL a
+NULL b
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_try_cast GROUP BY TRY_CAST(x AS INT);
+
+statement ok
+drop table t_try_cast;
Review Comment:
addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #26150:
URL: https://github.com/apache/datafusion/pull/26150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4228258449
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,52 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
Review Comment:
Could we also add a regression test for `TRY_CAST` over a `UNIQUE NOT NULL` key? For example, `('bad-a', 'b')` and `('bad-b', 'a')` both produce NULL when cast to INT, so the test could verify that `ORDER BY` retains `y`, `GROUP BY` preserves both groups, and selecting `y` when grouping only by the cast fails planning. This is optional coverage since the new helper already handles `TRY_CAST` conservatively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4271910807
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
Review Comment:
Added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4236255834
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
Review Comment:
`main` accepts this query and its result is correct. Rejecting it is the right call, but it is a user-facing change. Could you add it to "Are there any user-facing changes?" in the description? The upgrade guide already has it.
##########
datafusion/common/src/functional_dependencies.rs:
##########
@@ -480,24 +485,21 @@ pub fn aggregate_functional_dependencies(
{
// Indices into the GROUP BY list for this determinant:
let mut new_source_indices = vec![];
- let mut new_source_field_names = vec![];
- let source_field_names = source_indices
- .iter()
- .map(|&idx| &aggr_input_fields[idx])
- .collect::<Vec<_>>();
-
- for (idx, group_by_expr_name) in group_by_expr_names.iter().enumerate() {
- // When one of the input determinant expressions matches with
- // the GROUP BY expression, add the index of the GROUP BY
- // expression as a new determinant key:
- if source_field_names.contains(&group_by_expr_name) {
+ let mut new_source_input_indices = vec![];
+ for (idx, input_idx) in group_by_input_indices.iter().enumerate() {
+ // When one of the input determinant columns is a GROUP BY
+ // expression, add the index of the GROUP BY expression as a new
+ // determinant key:
+ if let Some(input_idx) = input_idx
+ && source_indices.contains(input_idx)
+ {
Review Comment:
A key column that is in the GROUP BY list two times loses its dependency. This is a regression from `main`, and only the builder / DataFrame API can reach it (SQL removes aliases from GROUP BY).
`.aggregate(vec![col("id"), col("state"), col("id").alias("k")], ...)` with `PRIMARY KEY (id)`:
| | Output dependency |
|---|---|
| `main` | `[0] -> [0, 1, 2]` |
| This PR | `[0, 1, 2] -> [0, 1, 2]` |
`id` and `id AS k` both map to input index 0, so the length check below fails. Results stay correct, but parent plans can no longer prune with `id`.
```suggestion
if let Some(input_idx) = input_idx
&& source_indices.contains(input_idx)
// A key column can be in the GROUP BY list more than one
// time (`x, x AS k`). Count it one time.
&& !new_source_input_indices.contains(&Some(*input_idx))
{
```
A unit test is optional here, because only the builder API reaches this. If you want one, this fails on this PR and passes with the suggestion:
<details><summary>Optional unit test for <code>plan.rs</code></summary>
```rust
#[test]
fn aggregate_group_by_repeated_key_keeps_key_dependency() -> Result<()> {
let constraints =
Constraints::new_unverified(vec![Constraint::PrimaryKey(vec![0])]);
let source = Arc::new(
LogicalTableSource::new(Arc::new(employee_schema()))
.with_constraints(constraints),
);
// `id` is in the GROUP BY list two times: as itself and as `k`.
let plan = LogicalPlanBuilder::scan("employee_csv", source, None)?
.aggregate(
vec![col("id"), col("state"), col("id").alias("k")],
Vec::<Expr>::new(),
)?
.build()?;
// `id` alone still determines the row.
let deps = plan.schema().functional_dependencies();
assert_eq!(deps.len(), 1);
assert_eq!(deps[0].source_indices, vec![0]);
assert_eq!(deps[0].target_indices, vec![0, 1, 2]);
Ok(())
}
```
</details>
With the suggestion, the full sqllogictest suite and the `datafusion-common`, `datafusion-expr` and `datafusion-optimizer` unit tests pass locally.
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
+
+statement ok
+drop table t_wide;
+
+# 6.6 The same with `TRY_CAST` over a `UNIQUE NOT NULL` key: 'bad-a' and
+# 'bad-b' both cast to NULL.
+statement ok
+CREATE TABLE t_try_cast (x VARCHAR NOT NULL UNIQUE, y VARCHAR) AS VALUES ('bad-a', 'b'), ('bad-b', 'a'), ('1', 'c');
+
+query TT
+SELECT x, y FROM t_try_cast ORDER BY TRY_CAST(x AS INT), y;
+----
+1 c
+bad-b a
+bad-a b
+
+query IT rowsort
+SELECT TRY_CAST(x AS INT) k, y FROM t_try_cast GROUP BY TRY_CAST(x AS INT), y;
+----
+1 c
+NULL a
+NULL b
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_try_cast GROUP BY TRY_CAST(x AS INT);
+
+statement ok
+drop table t_try_cast;
Review Comment:
The description says the bug also occurs with a key that comes from an inner GROUP BY, but no test covers that. The first two queries below return wrong results on `main`, and `main` accepts the third. All three pass on this PR.
```suggestion
drop table t_try_cast;
# 6.7 A key that comes from an inner GROUP BY: `x` is unique in the subquery,
# but `CAST(x AS INT)` is not.
statement ok
CREATE TABLE t_inner (x DOUBLE, y VARCHAR) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
query RT
SELECT x, y FROM (SELECT x, max(y) AS y FROM t_inner GROUP BY x) ORDER BY CAST(x AS INT), y;
----
1.2 a
1.1 b
2.1 c
query II rowsort
SELECT CAST(x AS INT) k, count(*) n FROM (SELECT x, max(y) AS y FROM t_inner GROUP BY x) GROUP BY CAST(x AS INT), y;
----
1 1
1 1
2 1
query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
SELECT y, count(*) FROM (SELECT x, max(y) AS y FROM t_inner GROUP BY x) GROUP BY CAST(x AS INT);
statement ok
drop table t_inner;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#issuecomment-6075307023
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26150?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `96.49123%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.76%. Comparing base ([`791660c`](https://app.codecov.io/gh/apache/datafusion/commit/791660cd54f0196978c98e9bca165c1beb7bbc78?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e0cbb0a`](https://app.codecov.io/gh/apache/datafusion/commit/e0cbb0af3477317ecf549a64b19aca21cc48c7e5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26150?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/expr/src/logical\_plan/plan.rs](https://app.codecov.io/gh/apache/datafusion/pull/26150?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | 85.71% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26150?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26150 +/- ##
==========================================
- Coverage 82.76% 82.76% -0.01%
==========================================
Files 1147 1147
Lines 450580 450563 -17
Branches 450580 450563 -17
==========================================
- Hits 372944 372921 -23
- Misses 54929 54932 +3
- Partials 22707 22710 +3
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26150?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4271887700
##########
datafusion/common/src/functional_dependencies.rs:
##########
@@ -480,24 +485,21 @@ pub fn aggregate_functional_dependencies(
{
// Indices into the GROUP BY list for this determinant:
let mut new_source_indices = vec![];
- let mut new_source_field_names = vec![];
- let source_field_names = source_indices
- .iter()
- .map(|&idx| &aggr_input_fields[idx])
- .collect::<Vec<_>>();
-
- for (idx, group_by_expr_name) in group_by_expr_names.iter().enumerate() {
- // When one of the input determinant expressions matches with
- // the GROUP BY expression, add the index of the GROUP BY
- // expression as a new determinant key:
- if source_field_names.contains(&group_by_expr_name) {
+ let mut new_source_input_indices = vec![];
+ for (idx, input_idx) in group_by_input_indices.iter().enumerate() {
+ // When one of the input determinant columns is a GROUP BY
+ // expression, add the index of the GROUP BY expression as a new
+ // determinant key:
+ if let Some(input_idx) = input_idx
+ && source_indices.contains(input_idx)
+ {
Review Comment:
Good catch, thanks. Applied your suggestion and added your unit test in b42b50ea96. The test fails without the change and passes with it.
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
+
+statement ok
+drop table t_wide;
+
+# 6.6 The same with `TRY_CAST` over a `UNIQUE NOT NULL` key: 'bad-a' and
+# 'bad-b' both cast to NULL.
+statement ok
+CREATE TABLE t_try_cast (x VARCHAR NOT NULL UNIQUE, y VARCHAR) AS VALUES ('bad-a', 'b'), ('bad-b', 'a'), ('1', 'c');
+
+query TT
+SELECT x, y FROM t_try_cast ORDER BY TRY_CAST(x AS INT), y;
+----
+1 c
+bad-b a
+bad-a b
+
+query IT rowsort
+SELECT TRY_CAST(x AS INT) k, y FROM t_try_cast GROUP BY TRY_CAST(x AS INT), y;
+----
+1 c
+NULL a
+NULL b
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_try_cast GROUP BY TRY_CAST(x AS INT);
+
+statement ok
+drop table t_try_cast;
Review Comment:
Added as 6.7 in b42b50ea96. The first two queries fail on `main`. The third one already fails planning on `main`, so it guards the rejection rather than reproducing the bug.
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
Review Comment:
Added a "Rejected queries" bullet to "Are there any user-facing changes?".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4271888501
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
Review Comment:
Added a "Rejected queries" bullet to "Are there any user-facing changes?".
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,89 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
+
+statement ok
+drop table t_cast;
+
+# 6.5 Only the key column itself carries the dependency, even when the cast
+# keeps distinct values distinct, like INT to BIGINT. PostgreSQL and DuckDB
+# reject this too.
+statement ok
+CREATE TABLE t_wide (x INT, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1, 'b'), (2, 'a');
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_wide GROUP BY CAST(x AS BIGINT);
+
+statement ok
+drop table t_wide;
+
+# 6.6 The same with `TRY_CAST` over a `UNIQUE NOT NULL` key: 'bad-a' and
+# 'bad-b' both cast to NULL.
+statement ok
+CREATE TABLE t_try_cast (x VARCHAR NOT NULL UNIQUE, y VARCHAR) AS VALUES ('bad-a', 'b'), ('bad-b', 'a'), ('1', 'c');
+
+query TT
+SELECT x, y FROM t_try_cast ORDER BY TRY_CAST(x AS INT), y;
+----
+1 c
+bad-b a
+bad-a b
+
+query IT rowsort
+SELECT TRY_CAST(x AS INT) k, y FROM t_try_cast GROUP BY TRY_CAST(x AS INT), y;
+----
+1 c
+NULL a
+NULL b
+
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_try_cast GROUP BY TRY_CAST(x AS INT);
+
+statement ok
+drop table t_try_cast;
Review Comment:
Added as 6.7 in b42b50ea96. The first two queries fail on `main`. The third one already fails planning on `main`, so it guards the rejection rather than reproducing the bug.
##########
datafusion/common/src/functional_dependencies.rs:
##########
@@ -480,24 +485,21 @@ pub fn aggregate_functional_dependencies(
{
// Indices into the GROUP BY list for this determinant:
let mut new_source_indices = vec![];
- let mut new_source_field_names = vec![];
- let source_field_names = source_indices
- .iter()
- .map(|&idx| &aggr_input_fields[idx])
- .collect::<Vec<_>>();
-
- for (idx, group_by_expr_name) in group_by_expr_names.iter().enumerate() {
- // When one of the input determinant expressions matches with
- // the GROUP BY expression, add the index of the GROUP BY
- // expression as a new determinant key:
- if source_field_names.contains(&group_by_expr_name) {
+ let mut new_source_input_indices = vec![];
+ for (idx, input_idx) in group_by_input_indices.iter().enumerate() {
+ // When one of the input determinant columns is a GROUP BY
+ // expression, add the index of the GROUP BY expression as a new
+ // determinant key:
+ if let Some(input_idx) = input_idx
+ && source_indices.contains(input_idx)
+ {
Review Comment:
Good catch, thanks. Applied your suggestion and added your unit test in b42b50ea96. The test fails without the change and passes with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4271135760
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,52 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
Review Comment:
Thanks! Added as 6.6 in 9d9f9f3805: `'bad-a'` and `'bad-b'` both cast to NULL, ORDER BY keeps `y`, GROUP BY keeps both NULL groups, and selecting `y` while grouping only by the cast fails planning.
##########
datafusion/expr/src/utils.rs:
##########
@@ -1020,6 +1020,20 @@ pub(crate) fn find_columns_referenced_by_expr(e: &Expr) -> Vec<Column> {
exprs
}
+/// Returns the index of the `schema` field that `expr` passes through
+/// unchanged, or `None` if `expr` computes a new value.
+///
+/// Only a column reference (possibly aliased) passes a field through. Use this
+/// instead of comparing [`Expr::schema_name`] with field names: different
+/// expressions can have the same name, e.g. `CAST(t.a AS INT)` is named `t.a`.
+pub fn passthrough_field_index(expr: &Expr, schema: &DFSchema) -> Option<usize> {
+ match expr {
+ Expr::Column(col) => schema.maybe_index_of_column(col),
Review Comment:
Fixed in 9d9f9f3805. `passthrough_field_index` now resolves the column with `qualified_field_from_column`, like any other column reference, so an ambiguous `id` over `a.id` and `b.id` passes nothing through. The unit test `passthrough_field_index_resolves_like_a_column_reference` covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4227070658
##########
datafusion/expr/src/utils.rs:
##########
@@ -1020,6 +1020,20 @@ pub(crate) fn find_columns_referenced_by_expr(e: &Expr) -> Vec<Column> {
exprs
}
+/// Returns the index of the `schema` field that `expr` passes through
+/// unchanged, or `None` if `expr` computes a new value.
+///
+/// Only a column reference (possibly aliased) passes a field through. Use this
+/// instead of comparing [`Expr::schema_name`] with field names: different
+/// expressions can have the same name, e.g. `CAST(t.a AS INT)` is named `t.a`.
+pub fn passthrough_field_index(expr: &Expr, schema: &DFSchema) -> Option<usize> {
+ match expr {
+ Expr::Column(col) => schema.maybe_index_of_column(col),
Review Comment:
`maybe_index_of_column` silently picks the first match for an unqualified column, unlike normal expression resolution, which rejects ambiguous references (or prefers the sole unqualified field). For a schema containing `a.id` and `b.id`, this reports `col("id")` as passing through `a.id`; callers can then apply that field's dependency and prune keys incorrectly. Resolve the column with the ambiguity-aware `qualified_field_from_column` first and return `None` when resolution fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#issuecomment-6093012950
Address all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#discussion_r4271135350
##########
datafusion/expr/src/utils.rs:
##########
@@ -1020,6 +1020,20 @@ pub(crate) fn find_columns_referenced_by_expr(e: &Expr) -> Vec<Column> {
exprs
}
+/// Returns the index of the `schema` field that `expr` passes through
+/// unchanged, or `None` if `expr` computes a new value.
+///
+/// Only a column reference (possibly aliased) passes a field through. Use this
+/// instead of comparing [`Expr::schema_name`] with field names: different
+/// expressions can have the same name, e.g. `CAST(t.a AS INT)` is named `t.a`.
+pub fn passthrough_field_index(expr: &Expr, schema: &DFSchema) -> Option<usize> {
+ match expr {
+ Expr::Column(col) => schema.maybe_index_of_column(col),
Review Comment:
Fixed in 9d9f9f3805. `passthrough_field_index` now resolves the column with `qualified_field_from_column`, like any other column reference, so an ambiguous `id` over `a.id` and `b.id` passes nothing through. The unit test `passthrough_field_index_resolves_like_a_column_reference` covers it.
##########
datafusion/sqllogictest/test_files/functional_dependencies.slt:
##########
@@ -358,6 +358,52 @@ drop table t_null;
statement ok
drop table t_probe;
+
+# 6. Only a column reference carries a dependency. `CAST(x AS INT)` is named
+# `t_cast.x`, like the PRIMARY KEY column, but it does not determine `y`:
+# 1.1 and 1.2 both cast to 1.
+statement ok
+CREATE TABLE t_cast (x DOUBLE, y VARCHAR, PRIMARY KEY (x)) AS VALUES (1.1, 'b'), (1.2, 'a'), (2.1, 'c');
+
+# 6.1 ORDER BY: `y` is the tie-breaker within `CAST(x AS INT) = 1` and is kept.
+query RT
+SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+1.2 a
+1.1 b
+2.1 c
+
+query TT
+EXPLAIN SELECT x, y FROM t_cast ORDER BY CAST(x AS INT), y;
+----
+logical_plan
+01)Sort: CAST(t_cast.x AS Int32) ASC NULLS LAST, t_cast.y ASC NULLS LAST
+02)--TableScan: t_cast projection=[x, y]
+
+# 6.2 GROUP BY: `y` is not selected but still splits the groups.
+query II rowsort
+SELECT CAST(x AS INT) k, count(*) n FROM t_cast GROUP BY CAST(x AS INT), y;
+----
+1 1
+1 1
+2 1
+
+# 6.3 GROUP BY output: the dependency `k -> y` does not exist, so `y` still
+# orders the rows within `k = 1`.
+query IT
+SELECT CAST(x AS INT) k, y FROM t_cast GROUP BY CAST(x AS INT), y ORDER BY k, y;
+----
+1 a
+1 b
+2 c
+
+# 6.4 `y` is not determined by the GROUP BY expression, so it can't be selected.
+query error DataFusion error: Error during planning: Column in SELECT must be in GROUP BY or an aggregate function
+SELECT y, count(*) FROM t_cast GROUP BY CAST(x AS INT);
Review Comment:
Thanks! Added as 6.6 in 9d9f9f3805: `'bad-a'` and `'bad-b'` both cast to NULL, ORDER BY keeps `y`, GROUP BY keeps both NULL groups, and selecting `y` while grouping only by the cast fails planning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match functional dependencies by column position, not by name [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26150:
URL: https://github.com/apache/datafusion/pull/26150#issuecomment-6173088354
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-common v55.1.0 (current)
Built [ 31.334s] (current)
Parsing datafusion-common v55.1.0 (current)
Parsed [ 0.066s] (current)
Building datafusion-common v55.1.0 (baseline)
Built [ 30.961s] (baseline)
Parsing datafusion-common v55.1.0 (baseline)
Parsed [ 0.068s] (baseline)
Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 1.125s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 65.023s] datafusion-common
Building datafusion-expr v55.1.0 (current)
error: running cargo-doc on crate 'datafusion-expr' failed with output:
-----
Compiling libc v0.2.190
Compiling proc-macro2 v1.0.107
Compiling unicode-ident v1.0.27
Compiling quote v1.0.47
Compiling libm v0.2.16
Compiling autocfg v1.5.1
Checking cfg-if v1.0.5
Compiling num-traits v0.2.19
Compiling syn v2.0.119
Checking memchr v2.8.3
Compiling serde_core v1.0.229
Compiling zerocopy v0.8.63
Compiling zmij v1.0.23
Compiling find-msvc-tools v0.1.14
Compiling shlex v2.0.1
Compiling serde_json v1.0.154
Checking itoa v1.0.18
Checking num-integer v0.1.47
Compiling jobserver v0.1.35
Compiling version_check v0.9.5
Compiling cc v1.8.0
Checking siphasher v1.0.4
Compiling getrandom v0.3.4
Checking bytes v1.12.1
Checking phf_shared v0.12.1
Compiling ahash v0.8.12
Checking num-bigint v0.5.1
Compiling chrono-tz v0.10.4
Checking equivalent v1.0.3
Checking foldhash v0.2.0
Checking iana-time-zone v0.1.65
Checking allocator-api2 v0.2.21
Checking hashbrown v0.17.1
Checking chrono v0.4.45
Checking arrow-schema v60.0.0
Checking phf v0.12.1
Checking once_cell v1.21.4
Checking num-complex v0.4.6
Checking lexical-util v1.0.7
Compiling pkg-config v0.3.34
Compiling syn v3.0.9
Compiling zerocopy-derive v0.8.63
Compiling zstd-sys v2.1.1+zstd.1.5.7
Checking lexical-parse-integer v1.0.6
Checking lexical-write-integer v1.0.6
Compiling semver v1.0.28
Compiling rustc_version v0.4.1
Checking lexical-write-float v1.0.6
Checking lexical-parse-float v1.0.6
Checking aho-corasick v1.1.5
Checking regex-syntax v0.8.11
Checking ryu v1.0.23
Checking half v2.7.1
Checking unicode-segmentation v1.13.3
Checking arrow-buffer v60.0.0
Checking unicode-width v0.2.2
Compiling zstd-safe v8.1.0
Checking comfy-table v8.0.1
Checking lexical-core v1.0.6
Checking arrow-data v60.0.0
Checking regex-automata v0.4.18
Compiling flatbuffers v25.12.19
Checking atoi v3.1.0
Checking base64 v0.23.1
Checking arrow-array v60.0.0
Checking csv-core v0.1.13
Compiling getrandom v0.4.3
Checking bitflags v2.13.2
Checking twox-hash v2.1.5
Checking lz4_flex v0.14.0
Checking csv v1.4.0
Checking regex v1.13.1
Checking indexmap v2.14.2
Checking simdutf8 v0.1.5
Compiling tokio-macros v2.7.2
Checking arrow-cmp v60.0.0
Checking arrow-arith v60.0.0
Checking arrow-select v60.0.0
Checking arrow-row v60.0.0
Checking pin-project-lite v0.2.17
Checking either v1.19.0
Compiling object v0.39.1
Checking log v0.4.34
Checking itertools v0.15.0
Checking tokio v1.53.3
Checking zstd v0.14.2
Checking uuid v1.28.0
Checking arrow-ord v60.0.0
Checking arrow-ipc v60.0.0
Checking arrow-cast v60.0.0
Checking arrow-string v60.0.0
Compiling parking_lot_core v0.9.12
Compiling anyhow v1.0.104
Checking smallvec v1.16.3
Checking arrow-csv v60.0.0
Checking arrow-json v60.0.0
Checking scopeguard v1.2.0
Checking lock_api v0.4.14
Compiling itertools v0.14.0
Checking arrow v60.0.0
Checking datafusion-common v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Compiling pin-project-internal v1.1.13
error[E0425]: cannot find value `sort_expr_names` in this scope
--> /home/runner/work/datafusion/datafusion/datafusion/common/src/functional_dependencies.rs:665:58
|
665 | let mut known_field_indices = HashSet::with_capacity(sort_expr_names.len());
| ^^^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find value `sort_expr_names` in this scope
--> /home/runner/work/datafusion/datafusion/datafusion/common/src/functional_dependencies.rs:666:61
|
666 | let mut required_sort_expr_indices = Vec::with_capacity(sort_expr_names.len());
| ^^^^^^^^^^^^^^^ not found in this scope
Compiling prost-derive v0.14.4
Checking pin-project v1.1.13
Checking parking_lot v0.12.5
Compiling stacker v0.1.25
Compiling serde v1.0.229
Compiling sqlparser_derive v0.6.0
Compiling serde_derive v1.0.229
Checking prost v0.14.4
Compiling recursive-proc-macro-impl v0.1.1
Compiling ar_archive_writer v0.5.3
Checking sqlparser v0.63.0
For more information about this error, try `rustc --explain E0425`.
error: could not compile `datafusion-common` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion-expr v55.1.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/expr --features default,proto,recursive_protection,sql,sqlparser &&
cargo check &&
cargo doc
Building datafusion-optimizer v55.1.0 (current)
error: running cargo-doc on crate 'datafusion-optimizer' failed with output:
-----
Compiling libc v0.2.190
Compiling proc-macro2 v1.0.107
Compiling unicode-ident v1.0.27
Compiling quote v1.0.47
Compiling autocfg v1.5.1
Compiling libm v0.2.16
Checking cfg-if v1.0.5
Compiling num-traits v0.2.19
Checking memchr v2.8.3
Compiling serde_core v1.0.229
Compiling syn v2.0.119
Compiling zerocopy v0.8.63
Compiling zmij v1.0.23
Compiling serde_json v1.0.154
Checking itoa v1.0.18
Compiling find-msvc-tools v0.1.14
Compiling shlex v2.0.1
Compiling jobserver v0.1.35
Compiling cc v1.8.0
Checking num-integer v0.1.47
Checking foldhash v0.2.0
Checking siphasher v1.0.4
Checking allocator-api2 v0.2.21
Compiling getrandom v0.3.4
Compiling version_check v0.9.5
Checking equivalent v1.0.3
Checking hashbrown v0.17.1
Compiling ahash v0.8.12
Checking phf_shared v0.12.1
Checking num-bigint v0.5.1
Checking bytes v1.12.1
Compiling chrono-tz v0.10.4
Checking iana-time-zone v0.1.65
Checking arrow-schema v60.0.0
Checking chrono v0.4.45
Checking phf v0.12.1
Compiling zerocopy-derive v0.8.63
Checking once_cell v1.21.4
Checking num-complex v0.4.6
Checking lexical-util v1.0.7
Compiling pkg-config v0.3.34
Checking lexical-write-integer v1.0.6
Checking lexical-parse-integer v1.0.6
Compiling semver v1.0.28
Compiling zstd-sys v2.1.1+zstd.1.5.7
Compiling rustc_version v0.4.1
Checking lexical-parse-float v1.0.6
Checking lexical-write-float v1.0.6
Compiling syn v3.0.9
Checking aho-corasick v1.1.5
Checking half v2.7.1
Checking unicode-width v0.2.2
Checking arrow-buffer v60.0.0
Compiling zstd-safe v8.1.0
Checking regex-syntax v0.8.11
Checking ryu v1.0.23
Checking unicode-segmentation v1.13.3
Checking arrow-data v60.0.0
Checking comfy-table v8.0.1
Checking lexical-core v1.0.6
Compiling flatbuffers v25.12.19
Checking atoi v3.1.0
Checking arrow-array v60.0.0
Checking base64 v0.23.1
Checking indexmap v2.14.2
Checking regex-automata v0.4.18
Checking csv-core v0.1.13
Compiling getrandom v0.4.3
Checking twox-hash v2.1.5
Checking bitflags v2.13.2
Checking lz4_flex v0.14.0
Checking csv v1.4.0
Checking arrow-cmp v60.0.0
Checking arrow-select v60.0.0
Checking simdutf8 v0.1.5
Checking arrow-row v60.0.0
Checking regex v1.13.1
Checking arrow-arith v60.0.0
Compiling tokio-macros v2.7.2
Checking arrow-ord v60.0.0
Checking arrow-string v60.0.0
Compiling parking_lot_core v0.9.12
Checking pin-project-lite v0.2.17
Checking either v1.19.0
Checking arrow-cast v60.0.0
Compiling object v0.39.1
Checking zstd v0.14.2
Checking itertools v0.15.0
Checking tokio v1.53.3
Checking arrow-ipc v60.0.0
Checking arrow-json v60.0.0
Checking arrow-csv v60.0.0
Checking uuid v1.28.0
Checking smallvec v1.16.3
Checking arrow v60.0.0
Checking scopeguard v1.2.0
Checking log v0.4.34
Checking lock_api v0.4.14
Checking datafusion-common v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Compiling pin-project-internal v1.1.13
Checking parking_lot v0.12.5
Compiling serde v1.0.229
error[E0425]: cannot find value `sort_expr_names` in this scope
--> /home/runner/work/datafusion/datafusion/datafusion/common/src/functional_dependencies.rs:665:58
|
665 | let mut known_field_indices = HashSet::with_capacity(sort_expr_names.len());
| ^^^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find value `sort_expr_names` in this scope
--> /home/runner/work/datafusion/datafusion/datafusion/common/src/functional_dependencies.rs:666:61
|
666 | let mut required_sort_expr_indices = Vec::with_capacity(sort_expr_names.len());
| ^^^^^^^^^^^^^^^ not found in this scope
Compiling serde_derive v1.0.229
Checking pin-project v1.1.13
Compiling stacker v0.1.25
Checking foldhash v0.1.5
Checking hashbrown v0.15.5
Compiling async-trait v0.1.92
Checking fixedbitset v0.5.7
Checking datafusion-doc v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Checking petgraph v0.8.3
For more information about this error, try `rustc --explain E0425`.
error: could not compile `datafusion-common` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion-optimizer v55.1.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/optimizer --features recursive_protection &&
cargo check &&
cargo doc
Building datafusion-sqllogictest v55.1.0 (current)
error: running cargo-doc on crate 'datafusion-sqllogictest' failed with output:
-----
Compiling proc-macro2 v1.0.107
Compiling unicode-ident v1.0.27
Compiling quote v1.0.47
Compiling libc v0.2.190
Checking cfg-if v1.0.5
Checking bytes v1.12.1
Checking memchr v2.8.3
Compiling serde_core v1.0.229
Compiling syn v3.0.9
Compiling syn v2.0.119
Compiling autocfg v1.5.1
Compiling shlex v2.0.1
Compiling find-msvc-tools v0.1.14
Checking itoa v1.0.18
Checking once_cell v1.21.4
Compiling jobserver v0.1.35
Checking foldhash v0.2.0
Checking allocator-api2 v0.2.21
Compiling cc v1.8.0
Checking equivalent v1.0.3
Compiling libm v0.2.16
Compiling num-traits v0.2.19
Checking hashbrown v0.17.1
Compiling zmij v1.0.23
Checking pin-project-lite v0.2.17
Checking futures-core v0.3.34
Compiling zerocopy v0.8.63
Checking indexmap v2.14.2
Compiling zerocopy-derive v0.8.63
Checking futures-sink v0.3.34
Checking errno v0.3.14
Checking signal-hook-registry v1.4.8
Compiling tokio-macros v2.7.2
Checking num-integer v0.1.47
Checking socket2 v0.6.5
Checking mio v1.2.4
Compiling version_check v0.9.5
Compiling serde v1.0.229
Checking tokio v1.53.3
Compiling serde_json v1.0.154
Compiling serde_derive v1.0.229
Checking slab v0.4.12
Compiling getrandom v0.3.4
Checking futures-channel v0.3.34
Checking smallvec v1.16.3
Compiling futures-macro v0.3.34
Compiling synstructure v0.14.0
Checking http v1.5.0
Checking futures-io v0.3.34
Checking futures-task v0.3.34
Checking iana-time-zone v0.1.65
Checking chrono v0.4.45
Checking futures-util v0.3.34
Checking siphasher v1.0.4
Compiling zerofrom-derive v0.1.8
Compiling tracing-attributes v0.1.31
Checking tracing-core v0.1.36
Checking zerofrom v0.1.8
Checking tracing v0.1.44
Compiling yoke-derive v0.8.4
Checking num-complex v0.4.6
Compiling getrandom v0.4.3
Checking stable_deref_trait v1.2.1
Checking rand_core v0.10.1
Checking cpufeatures v0.3.1
Checking half v2.7.1
Checking phf_shared v0.12.1
Compiling ahash v0.8.12
Checking num-bigint v0.5.1
Compiling zerovec-derive v0.11.6
Checking getrandom v0.2.17
Compiling pkg-config v0.3.34
Checking yoke v0.8.3
Compiling chrono-tz v0.10.4
Checking phf v0.12.1
Checking arrow-schema v60.0.0
Compiling displaydoc v0.2.7
Checking arrow-buffer v60.0.0
Checking zerovec v0.11.8
Checking percent-encoding v2.3.2
Compiling thiserror v2.0.21
Checking arrow-data v60.0.0
Compiling thiserror-impl v2.0.21
Compiling ring v0.17.14
Checking bitflags v2.13.2
Checking chacha20 v0.10.2
Compiling semver v1.0.28
Checking log v0.4.34
Checking rand v0.10.3
Checking tinystr v0.8.4
Checking untrusted v0.9.0
Checking writeable v0.6.4
Checking base64 v0.23.1
Checking litemap v0.8.3
Checking icu_locale_core v2.3.0
Checking potential_utf v0.1.6
Checking zerotrie v0.2.5
Compiling zstd-sys v2.1.1+zstd.1.5.7
Compiling async-trait v0.1.92
Compiling icu_properties_data v2.3.0
Compiling icu_normalizer_data v2.3.0
Checking utf8_iter v1.0.4
Checking icu_collections v2.3.0
Checking icu_provider v2.3.1
Checking tokio-util v0.7.20
Checking aho-corasick v1.1.5
Checking regex-syntax v0.8.11
Compiling object v0.39.1
Compiling zstd-safe v8.1.0
Checking ryu v1.0.23
Checking lexical-util v1.0.7
Checking arrow-array v60.0.0
Checking icu_normalizer v2.3.0
Checking icu_properties v2.3.0
Checking regex-automata v0.4.18
Checking either v1.19.0
Compiling rustix v1.1.5
Checking idna_adapter v1.2.2
Checking arrow-cmp v60.0.0
Checking arrow-select v60.0.0
Checking regex v1.13.1
Checking form_urlencoded v1.2.2
Compiling parking_lot_core v0.9.12
Checking typenum v1.20.1
Compiling crc32fast v1.5.2
Checking unicode-width v0.2.2
Checking idna v1.1.0
Checking lexical-parse-integer v1.0.6
Checking lexical-write-integer v1.0.6
Compiling rustc_version v0.4.1
Checking futures-executor v0.3.34
Compiling pin-project-internal v1.1.13
Checking simd-adler32 v0.3.10
Checking adler2 v2.0.1
Checking scopeguard v1.2.0
Checking lock_api v0.4.14
Checking miniz_oxide v0.9.1
Compiling flatbuffers v25.12.19
Checking futures v0.3.34
Checking lexical-write-float v1.0.6
Checking pin-project v1.1.13
Checking lexical-parse-float v1.0.6
Checking url v2.5.8
Compiling ar_archive_writer v0.5.3
Checking zlib-rs v0.6.8
Checking unicode-segmentation v1.13.3
Compiling cfg_aliases v0.2.2
Compiling nix v0.31.3
Compiling psm v0.1.32
Checking comfy-table v8.0.1
Checking lexical-core v1.0.6
Checking arrow-ord v60.0.0
Checking twox-hash v2.1.5
Checking flate2 v1.1.10
Compiling stacker v0.1.25
Checking atoi v3.1.0
Compiling snap v1.1.2
Checking hex v0.4.3
Checking alloc-no-stdlib v3.0.0
Checking alloc-stdlib v0.3.0
Checking arrow-cast v60.0.0
Checking lz4_flex v0.14.0
Checking parking_lot v0.12.5
Checking csv-core v0.1.13
Checking simdutf8 v0.1.5
Checking same-file v1.0.6
Checking humantime v2.4.0
Checking walkdir v2.5.0
Checking zstd v0.14.2
Checking csv v1.4.0
Checking arrow-ipc v60.0.0
Checking brotli-decompressor v6.0.1
Checking itertools v0.15.0
Compiling recursive-proc-macro-impl v0.1.1
Checking recursive v0.1.1
Checking arrow-json v60.0.0
Checking brotli v9.0.0
Checking arrow-csv v60.0.0
Checking arrow-string v60.0.0
Checking arrow-row v60.0.0
Checking arrow-arith v60.0.0
Checking object_store v0.14.2
Checking uuid v1.28.0
Compiling sqlparser_derive v0.6.0
Compiling seq-macro v0.3.6
Checking arrow v60.0.0
Checking sqlparser v0.63.0
Checking hybrid-array v0.4.15
Checking ppv-lite86 v0.2.21
Checking cmov v0.5.4
Checking block-buffer v0.12.1
Checking parquet v60.0.0
Checking ctutils v0.4.3
Checking crypto-common v0.2.2
Checking linux-raw-sys v0.12.1
Checking const-oid v0.10.2
Checking digest v0.11.3
Checking rand_core v0.9.5
Checking rand_chacha v0.9.0
Checking datafusion-doc v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Checking rand v0.9.5
Compiling crossbeam-utils v0.8.23
Checking foldhash v0.1.5
Checking hashbrown v0.15.5
Checking fastrand v2.5.0
Checking fixedbitset v0.5.7
Compiling heck v0.5.0
Checking petgraph v0.8.3
Checking tempfile v3.27.0
Checking md-5 v0.11.0
Checking sha2 v0.11.1
Checking hashbrown v0.14.5
Checking dashmap v6.2.1
Compiling datafusion-macros v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Compiling blake3 v1.8.7
Compiling anyhow v1.0.104
Checking constant_time_eq v0.4.2
Checking arrayvec v0.7.8
Compiling itertools v0.14.0
Checking blake2 v0.11.0
Compiling liblzma-sys v0.4.9
Checking libbz2-rs-sys v0.2.5
Compiling prost-derive v0.14.4
Checking bzip2 v0.6.1
Checking datafusion-common-runtime v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Checking compression-core v0.4.33
Compiling httparse v1.10.1
Checking http-body v1.1.0
Checking base64 v0.22.1
Checking glob v0.3.4
Checking tower-service v0.3.3
Checking atomic-waker v1.1.2
Checking try-lock v0.2.5
Checking fnv v1.0.7
Checking h2 v0.4.20
Checking liblzma v0.4.8
Checking compression-codecs v0.4.45
Checking async-compression v0.4.50
Checking want v0.3.2
Checking tokio-stream v0.1.19
Checking httpdate v1.0.3
Checking zeroize v1.9.1
Checking rustls-pki-types v1.15.1
Checking hyper v1.12.0
Checking hyper-util v0.1.21
Compiling prost v0.14.4
Checking datafusion-common v55.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
error[E0425]: cannot find value `sort_expr_names` in this scope
--> /home/runner/work/datafusion/datafusion/datafusion/common/src/functional_dependencies.rs:665:58
|
665 | let mut known_field_indices = HashSet::with_capacity(sort_expr_names.len());
| ^^^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find value `sort_expr_names` in this scope
--> /home/runner/work/datafusion/datafusion/datafusion/common/src/functional_dependencies.rs:666:61
|
666 | let mut required_sort_expr_indices = Vec::with_capacity(sort_expr_names.len());
| ^^^^^^^^^^^^^^^ not found in this scope
Checking http-body-util v0.1.5
Checking num-bigint v0.4.8
Compiling rustls v0.23.46
Compiling prettyplease v0.2.37
Checking tower-layer v0.3.3
Checking sync_wrapper v1.0.2
Compiling prost-types v0.14.4
Checking rustls-webpki v0.103.15
Compiling serde_derive_internals v0.29.1
Compiling schemars v0.8.22
Checking subtle v2.6.1
For more information about this error, try `rustc --explain E0425`.
error: could not compile `datafusion-common` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion-sqllogictest v55.1.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/sqllogictest --features avro,backtrace,bytes,chrono,datafusion-substrait,parquet_encryption,postgres,postgres-types,substrait,testcontainers-modules,tokio-postgres &&
cargo check &&
cargo doc
error: aborting due to failure to build rustdoc for crate datafusion-expr v55.1.0
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org