You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Yacobolo (via GitHub)" <gi...@apache.org> on 2023/10/06 20:22:09 UTC
[I] Support for WITH Statements in DataFusion [arrow-datafusion]
Yacobolo opened a new issue, #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760
### Is your feature request related to a problem or challenge?
I was trying to write an adapter for dbt, but ran into a problem with it not supporting "with"statments.
### Describe the solution you'd like
be able to do:
```sql
WITH preprocessed_data AS (
SELECT id, name, SUM(sales) as total_sales
FROM sales_data
WHERE date > '2022-01-01'
GROUP BY id, name
)
SELECT name, total_sales
FROM preprocessed_data
WHERE total_sales > 1000;
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion (they are currently silently ignored) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760#issuecomment-1751513222
> Sorry for the false claim. I couldn't find it in the docs and I had a syntax error on my end!
No worries -- any chance you can add it to the docs @Yacobolo 🙏
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion (they are currently silently ignored) [arrow-datafusion]
Posted by "Yacobolo (via GitHub)" <gi...@apache.org>.
Yacobolo commented on issue #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760#issuecomment-1751445491
Sorry for the false claim. I couldn't find it in the docs and I had a syntax error on my end!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion (they are currently silently ignored) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760#issuecomment-1751418389
```
❯ create table table1 (a int) as values (1);
0 rows in set. Query took 0.002 seconds.
❯ WITH table2 AS (
SELECT a, count(1) as cnt
FROM table1
GROUP BY A
)
SELECT *
FROM table2
WHERE cnt>0
LIMIT 10;
+---+-----+
| a | cnt |
+---+-----+
| 1 | 1 |
+---+-----+
```
🤔 you are right @devinjdangelo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion (they are currently silently ignored) [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760#issuecomment-1751419505
```
❯ create table t(x int) as values (1), (2), (3), (2), (5), (1);
0 rows in set. Query took 0.004 seconds.
❯ with t2 as (select * from t where x > 2) select * **from t2**;
Error during planning: SELECT * with no tables specified is not valid
```
I think you were just missing from t2 in your original example @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion (they are currently silently ignored) [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760#issuecomment-1751415697
I might be missing something, but I thought Datafusion did support [with statements](https://arrow.apache.org/datafusion/user-guide/sql/select.html)
I tested this query locally and it worked:
```sql
WITH table2 AS (
SELECT a, count(1) as cnt
FROM table1
GROUP BY A
)
SELECT *
FROM table2
WHERE cnt>10
LIMIT 10
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion (they are currently silently ignored) [arrow-datafusion]
Posted by "Yacobolo (via GitHub)" <gi...@apache.org>.
Yacobolo closed issue #7760: Support for WITH Statements in DataFusion (they are currently silently ignored)
URL: https://github.com/apache/arrow-datafusion/issues/7760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for WITH Statements in DataFusion [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7760:
URL: https://github.com/apache/arrow-datafusion/issues/7760#issuecomment-1751400922
Thanks @Yacobolo -- I think this would be a relatively straightforward feature to add to the SQL planner
It actually looks like the sql parser already supports this and DataFusion is just ignoring it completely
```
❯ create table t(x int) as values (1), (2), (3), (2), (5), (1);
0 rows in set. Query took 0.004 seconds.
❯ with t2 as (select * from t where x > 2) select *;
Error during planning: SELECT * with no tables specified is not valid
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org