You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/09/11 11:55:54 UTC
[PR] Fix regression in SELECT FROM syntax with WHERE clause [datafusion]
adriangb opened a new pull request, #17520:
URL: https://github.com/apache/datafusion/pull/17520
## Summary
- Fixes #17513
- Fixes regression introduced by #17295 where `SELECT FROM table WHERE condition` incorrectly returned all columns instead of empty projection
## Problem
PR #17295 added support for FROM-first syntax (e.g., `FROM table` without SELECT). However, it couldn't distinguish between:
- `FROM table` - should select all columns (FROM-first syntax)
- `SELECT FROM table WHERE ...` - should select no columns (empty projection)
Both cases resulted in empty projection lists in the parser, making them indistinguishable.
## Solution
The fix checks for the presence of a WHERE clause to differentiate:
- Empty projection + no WHERE clause = FROM-first syntax → add wildcard projection
- Empty projection + WHERE clause = SELECT FROM syntax → keep empty projection
This ensures:
- `FROM table` returns all columns (expected FROM-first behavior)
- `SELECT FROM table WHERE condition` returns empty columns but correct row count
- The optimizer still pushes down column requirements to TableScan for efficiency
## Test Changes
Updated the test expectation in `projection.slt` to correctly show that `SELECT FROM table WHERE ...` produces an empty Projection node in the logical plan, while the TableScan still only projects the necessary columns.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <no...@anthropic.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression in SELECT FROM syntax with WHERE clause [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#discussion_r2340467298
##########
datafusion-testing:
##########
Review Comment:
Will fix before merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3286713964
@simonvandel I'm sorry we had to revert your contribution. I hope you're able to contribute again and now that we have the regression test and clarity about the behavior it should be okay to re-add your feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3283824114
> * We want the opposite behavior between them (agreed?)
agreed.
BTW Trino has the following syntax:
```sql
-- equivalent of SELECT * FROM my_table
TABLE my_table;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3285898307
Thank you @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression in SELECT FROM syntax with WHERE clause [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3280334178
Hmm good point. I agree with you. And actually `select a from t1 where t1.a > 1;` and `select b from t1 where t1.a > 1;` work as expected.
@xudong963 I think your original report may be a non issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3283255062
Need to revert doc changes too:
https://github.com/apache/datafusion/blob/351675ddc27c42684a079b3a89fe2dee581d89a2/docs/source/user-guide/sql/select.md?plain=1#L78-L90
- See https://github.com/apache/datafusion/pull/17295/files#diff-a957e8984c4b0756fc675eb0979112487a1aca8a27f04c67e109e3c15a254ec3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3287777761
Thank you all! Sorry, I didn't get a chance to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression: SELECT FROM syntax should return empty projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3281326431
Yeah it seems like what this is coming to is:
- We can't differentiate between `SELECT FROM t;` and `FROM t;`
- We want the opposite behavior between them (agreed?)
- Thus we need to revert https://github.com/apache/datafusion/pull/17295 and add a regression test
I'd like to get some input from the authors of https://github.com/apache/datafusion/pull/17295 before we proceed down that path. Then I agree @findepi we can add a better regression test that asserts against the relevant bit of the plan in Rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#discussion_r2341521150
##########
datafusion/sqllogictest/test_files/projection.slt:
##########
@@ -252,3 +252,30 @@ physical_plan
statement ok
drop table t;
+
+# Regression test for 17513
Review Comment:
```suggestion
# Regression test for
# https://github.com/apache/datafusion/issues/17513
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression in SELECT FROM syntax with WHERE clause [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3280308420
I wasn't aware we allowed `SELECT FROM t1 WHERE ...` syntax, omitting columns entirely. I checked out your branch and observed this behaviour which doesn't seem consistent:
```sql
> select from t1;
+---+---+
| a | b |
+---+---+
| 1 | 2 |
+---+---+
1 row(s) fetched.
Elapsed 0.007 seconds.
> select from t1 where 1=0;
0 row(s) fetched.
Elapsed 0.006 seconds.
```
- Using the same `t1` table as from the added slt test
So `SELECT FROM t1` would return all columns but adding a `WHERE` clause now omits all columns? Is this intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3286659694
This one looks good thanks everyone!
I also made a backport PR to `branch-50` branch so we can include it in the RC
- https://github.com/apache/datafusion/pull/17544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #17520:
URL: https://github.com/apache/datafusion/pull/17520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3285732843
> Need to revert doc changes too:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression in SELECT FROM syntax with WHERE clause [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3281071460
cc @simonvandel and @Jefffrey who were part of the original PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Revert #17295 (Support from-first SQL syntax) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#discussion_r2341523858
##########
datafusion/core/tests/sql/select.rs:
##########
@@ -344,3 +344,27 @@ async fn test_version_function() {
assert_eq!(version.value(0), expected_version);
}
+
+/// Regression test for https://github.com/apache/datafusion/issues/17513
+/// See https://github.com/apache/datafusion/pull/17520
+#[tokio::test]
+async fn test_select_no_projection() -> Result<()> {
+ let tmp_dir = TempDir::new()?;
+ let ctx = create_ctx_with_partition(&tmp_dir, 1).await?;
+
+ let results = ctx.sql("SELECT FROM test").await?.collect().await?;
+ // We should get all of the rows, just without any columns
+ let total_rows: usize = results.iter().map(|b| b.num_rows()).sum();
+ assert_eq!(total_rows, 10); // `create_ctx_with_partition` creates 10 rows per partition and we chose 1 partition
Review Comment:
```suggestion
// `create_ctx_with_partition` creates 10 rows per partition and we chose 1 partition
let ctx = create_ctx_with_partition(&tmp_dir, 1).await?;
let results = ctx.sql("SELECT FROM test").await?.collect().await?;
// We should get all of the rows, just without any columns
let total_rows: usize = results.iter().map(|b| b.num_rows()).sum();
assert_eq!(total_rows, 10);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression in SELECT FROM syntax with WHERE clause [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#issuecomment-3280234425
@simonvandel and @Jefffrey could you review please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix regression: SELECT FROM syntax should return empty projection [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #17520:
URL: https://github.com/apache/datafusion/pull/17520#discussion_r2341308087
##########
datafusion/sqllogictest/test_files/projection.slt:
##########
@@ -252,3 +252,30 @@ physical_plan
statement ok
drop table t;
+
+# Regression test for 17513
+
+query I
+COPY (select 1 as a, 2 as b)
+TO 'test_files/scratch/projection/17513.parquet'
+STORED AS PARQUET;
+----
+1
+
+statement ok
+create external table t1 stored as parquet location 'test_files/scratch/projection/17513.parquet';
+
+query TT
+explain format indent
+select from t1 where t1.a > 1;
Review Comment:
Plan tests are cool, but plan changes are easy to overlook. They need manual verification.
Query results-based tests are less disputable.
Can we also run this query and assert the result?
sqllogic doesn't support this (maybe we would need to change the rendering or result lines to make them non-empty).
If slt doesn't allow testing this, then we need a test in Rust.
##########
datafusion/sql/src/select.rs:
##########
@@ -676,13 +676,12 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
let mut prepared_select_exprs = vec![];
let mut error_builder = DataFusionErrorBuilder::new();
- // Handle the case where no projection is specified but we have a valid FROM clause
- // In this case, implicitly add a wildcard projection (SELECT *)
- let projection = if projection.is_empty() && !empty_from {
- vec![SelectItem::Wildcard(WildcardAdditionalOptions::default())]
- } else {
- projection
- };
+ // Note: We previously added wildcard projection for empty projections to support
+ // FROM-first syntax, but this caused a regression where "SELECT FROM table"
+ // incorrectly returned all columns instead of empty projection.
+ // For now, we don't add wildcard, which fixes the regression but breaks FROM-first.
+ // A proper fix would need to distinguish between "FROM table" and "SELECT FROM table"
+ // at the parser level, which sqlparser currently doesn't support.
Review Comment:
I am not sure the comment is needed. The test is an automated comment.
This PR is logically a revert + regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org