You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "berkaysynnada (via GitHub)" <gi...@apache.org> on 2025/02/05 13:22:09 UTC
[I] Add Support for Dynamic SQL Macros for Flexible Column Selection [datafusion]
berkaysynnada opened a new issue, #14512:
URL: https://github.com/apache/datafusion/issues/14512
### Is your feature request related to a problem or challenge?
People may need to write flexible SQL queries that adapt to different table schemas without hardcoding column names.
Query reusability is very limited currently, and creating dynamic queries requires external templating. A built-in mechanism to define and use dynamic SQL macros would significantly enhance the expressiveness and reusability
### Describe the solution you'd like
Introduce support for dynamic SQL macros that allow users to parameterize column selection. For example, a macro could accept lists of columns to include or exclude and then automatically select the appropriate columns from a given table or CTE. An example usage might look like this:
```sql
CREATE OR REPLACE MACRO dynamic_select(
include_cols,
exclude_cols
) AS TABLE (
FROM source_table
SELECT
COLUMNS(col ->
(list_contains(include_cols, col))
AND
(NOT list_contains(exclude_cols, col))
)
);
WITH source_table AS (
SELECT * FROM employees
)
FROM dynamic_select(
['id', 'name', 'department'],
['salary']
);
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add Support for Dynamic SQL Macros for Flexible Column Selection [datafusion]
Posted by "khushishukla2813 (via GitHub)" <gi...@apache.org>.
khushishukla2813 commented on issue #14512:
URL: https://github.com/apache/datafusion/issues/14512#issuecomment-2703692801
Hey @berkaysynnada Im interested in this issue and wanna participate in GSoC. Before getting started, I wanted to clarify if a column is present in both include_cols and exclude_cols how would prioritization happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add Support for Dynamic SQL Macros for Flexible Column Selection [datafusion]
Posted by "buraksenn (via GitHub)" <gi...@apache.org>.
buraksenn commented on issue #14512:
URL: https://github.com/apache/datafusion/issues/14512#issuecomment-4030666030
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add Support for Dynamic SQL Macros for Flexible Column Selection [datafusion]
Posted by "kumarlokesh (via GitHub)" <gi...@apache.org>.
kumarlokesh commented on issue #14512:
URL: https://github.com/apache/datafusion/issues/14512#issuecomment-2816854617
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add Support for Dynamic SQL Macros for Flexible Column Selection [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on issue #14512:
URL: https://github.com/apache/datafusion/issues/14512#issuecomment-2704290345
> where to store macro definitions in memory, catalog, or elsewhere
> how macros gonna integrate into query execution
These questions are not easy to answer straightforwardly at the moment. Once the project assignments are done, we will focus on these questions during the first week and likely discuss them with the entire community. However, if you develop some intuition and can make proposals, it will certainly be beneficial.
> should I modify parser.rs to support CREATE MACRO
Yeah, you probably should
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add Support for Dynamic SQL Macros for Flexible Column Selection [datafusion]
Posted by "buraksenn (via GitHub)" <gi...@apache.org>.
buraksenn commented on issue #14512:
URL: https://github.com/apache/datafusion/issues/14512#issuecomment-5064322420
untake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org