You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "findepi (via GitHub)" <gi...@apache.org> on 2024/08/24 06:34:53 UTC
[I] List available functions (`SHOW FUNCTIONS`) [datafusion]
findepi opened a new issue, #12144:
URL: https://github.com/apache/datafusion/issues/12144
### Is your feature request related to a problem or challenge?
I as a user would want to see a list of available functions.
### Describe the solution you'd like
```sql
SHOW FUNCTIONS [ LIKE <pattern> ];
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2456167400
I created https://github.com/apache/datafusion/pull/13255 for the `routines` table. Just organize the related works here:
- [ ] https://github.com/apache/datafusion/pull/13255
- [ ] Implement `parameters` table
- [ ] Enhance the type inferring from the function signature
- [ ] https://github.com/apache/datafusion/pull/12266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2334139425
We could add a `datafusion_functions` table function in datafusion-cli or other downstream implementation perhaps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2323431247
> We could not only get the function name but also `function_type`, `return_type`, `parameters` ... and so on
absolutely, more info than just function name is a good thing to have
> I think we should support table function similar to https://duckdb.org/docs/sql/functions/overview.html
The SQL standard way of presenting function information seems to be via `information_schema.routines` view. This is similar to how tables' information is queryable. But then we still have SHOW TABLES on top of `information_schema.tables` for easier access.
We can certainly present this information as a table function (instead of, or additionally to).
What would be the main advantage of doing so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2447197521
> We are pretty close to completing the project to have the documentation programatically available
>
> * [[EPIC] Automatically generate all function documentation from code #12740](https://github.com/apache/datafusion/issues/12740)
> I think this wuold be straightforward to add now
I checked the documents. It's very good for human reading but I think it doesn't have the type information (parameters and return_type). They're important for the third-party tool to integrate with DataFusion-based applications.
Following the `information_schema` way, I think we need [routines](https://www.postgresql.org/docs/current/infoschema-routines.html) and [parameters](https://www.postgresql.org/docs/current/infoschema-parameters.html) (They're the Postgres tables but I think we don't need to implement all the column of theirs).
@findepi are you still working on this?
I would like to push this feature. If you don't have bandwidth, I could help to implement the information_schema parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2334137732
It is my opinion we should follow the same pattern as implemented for `SHOW TABLES` as it seems to work well and allows for easier integration into third-party tools (like BI tools) that may query such tables for information.
Specifically, I think that means:
1. implement the `information_schema` schema (potentially with some additional custom columns for DataFusion like `help_text`), etc
2. Implement `SHOW FUNCTIONS` as a query on top of the information schema
3. (maybe this is wishful thinking) automatically create function documentation by scraping the code (as we do for configuration values)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2449440847
I paused this for now. @goldmedal go for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2323098476
I think we should support table function similar to https://duckdb.org/docs/sql/functions/overview.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2442034262
Is there any update on this issue? In my downstream project, Wren AI, we would register various custom functions (UDF, UDAF, etc.) dynamically. This feature would be useful in our scenario for getting the available functions in the current session.
Currently, I can only access functions through `SessionState::scalar_functions()` or similar methods. However, obtaining parameter type or return type information is challenging.
As @alamb suggested, I think we can begin by implementing `information_schema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12144: List available functions (`SHOW FUNCTIONS`)
URL: https://github.com/apache/datafusion/issues/12144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2323576458
> We can certainly present this information as a table function (instead of, or additionally to).
What would be the main advantage of doing so?
We could easily apply any query on the table function like
```
SELECT DISTINCT ON(function_name)
function_name,
function_type,
return_type,
parameters,
parameter_types,
description
FROM duckdb_functions()
WHERE function_type = 'scalar'
AND function_name LIKE 'b%'
ORDER BY function_name;
```
and we can get a subset of information with `select`
Unlike the syntax here, it is completely different so we need additional syntax support. It is also not easy to select subset of information
```
SHOW FUNCTIONS [ LIKE '<pattern>' ]
[ IN
{
ACCOUNT |
CLASS <class_name> |
DATABASE |
DATABASE <database_name> |
SCHEMA |
SCHEMA <schema_name> |
<schema_name>
}
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2326245954
> We could easily apply any query on the table function like
This is a good point.
Would we get similar benefits if we implemented `information_schema.routines` view (which SQL spec happens to define)?
In any case, writing a full query to `information_schema.*` or to a table function, is a few key strokes. That's probably why we have `SHOW TABLES` statement even though the information is also available via `information_schema.tabes` and is more queryable there (one can select desired columns or apply filters).
Summing up we have discussed these implementation options
1. `information_schema.routines` (as defined by the SQL spec)
2. `SHOW FUNCTIONS` (similar to `SHOW TABLES` and similar (but not identical) to some query engines like Snowflake, Trino/Presto or MySQL)
3. `datafusion_functions()` table function (similar to DuckDB's `duckdb_functions()`)
Which one or which ones do we want to implement?
How do arrive at this decision to make sure it's not questioned later by some project maintainer(s) that I failed to involve in the discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2326312484
I like the idea of `datafusion_functions()` as a `table`. I also prefer existing system (Postgres, DuckDB is my preference) than spec.
> How do arrive at this decision to make sure it's not questioned later by some project maintainer(s) that I failed to involve in the discussion?
We could wait for a few days for more opionion. Don't worry about being questioned later on. Datafusion is a continuously evolving and breaking change friendly project, if there comes out a better idea we could review and even change it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2442556156
We are pretty close to completing the project to have the documentation programatically available
- https://github.com/apache/datafusion/issues/12740
I think this wuold be straightforward to add now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] List available functions (`SHOW FUNCTIONS`) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12144:
URL: https://github.com/apache/datafusion/issues/12144#issuecomment-2343424665
> (maybe this is wishful thinking) automatically create function documentation by scraping the code (as we do for configuration values)
I filed https://github.com/apache/datafusion/issues/12432 to track this idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org