You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/07/10 07:10:02 UTC
[PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
zhuqi-lucas opened a new pull request, #23438:
URL: https://github.com/apache/datafusion/pull/23438
# Which issue does this PR close?
Closes #23437.
# Rationale for this change
`information_schema.routines` and `SHOW FUNCTIONS` currently enumerate only scalar / aggregate / window UDFs. Table functions (UDTFs) registered via `SessionContext::register_udtf` are omitted, making them undiscoverable through SQL. Discovery matters because downstream tooling (DataFusion CLI, Massive's atlas SQL surface, dbt-datafusion, etc.) uses these SQL surfaces to list available functions.
# What changes are included in this PR?
**1. Snapshot table functions into the information_schema provider.**
`TaskContext` cannot expose table_functions without introducing a crate cycle (`TableFunction` lives in `datafusion-catalog`, which depends on `datafusion-execution` where `TaskContext` lives). Instead, snapshot `SessionState.table_functions` into `InformationSchemaProvider` at construction time. The provider is built per-query via `schema_for_ref`, so the snapshot stays fresh.
**2. Emit UDTF rows.**
- `make_routines`: one row per UDTF with `routine_type = \"FUNCTION\"`, `function_type = \"TABLE\"`, `data_type = \"TABLE\"`.
- `make_parameters`: a single synthetic `OUT` row per UDTF (`data_type = \"TABLE\"`) so the JOIN in `SHOW FUNCTIONS` resolves. UDTFs have no scalar signature, so no `IN` rows are emitted.
**3. Rewrite the SHOW FUNCTIONS SQL.**
The old query joined `parameters p (INNER)` requiring both IN and OUT rows before joining `routines`. UDTFs have no IN parameters, so they fell out. New shape:
\`\`\`
FROM (parameters WHERE mode='OUT') o
LEFT JOIN (parameters WHERE mode='IN') i
ON matching keys
JOIN information_schema.routines r
ON name matches
\`\`\`
# Are these changes tested?
Verified locally by registering a UDTF and running SHOW FUNCTIONS:
\`\`\`
> SHOW FUNCTIONS LIKE 'stale%'
+---------------+-------------+---------------+
| function_name | return_type | function_type |
| stale_files | TABLE | TABLE |
+---------------+-------------+---------------+
\`\`\`
Happy to add sqllogictests in `datafusion/sqllogictest/test_files/information_schema.slt` if maintainers prefer.
# Are there any user-facing changes?
Yes: SHOW FUNCTIONS output now includes table functions. `information_schema.routines` and `information_schema.parameters` gain rows for each registered UDTF. Existing rows are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #23438:
URL: https://github.com/apache/datafusion/pull/23438#discussion_r3592437393
##########
datafusion/catalog/src/information_schema.rs:
##########
@@ -81,14 +82,28 @@ impl InformationSchemaProvider {
/// Creates a new [`InformationSchemaProvider`] for the provided `catalog_list`
pub fn new(catalog_list: Arc<dyn CatalogProviderList>) -> Self {
Self {
- config: InformationSchemaConfig { catalog_list },
+ config: InformationSchemaConfig {
+ catalog_list,
+ table_functions: HashMap::new(),
+ },
}
}
+
+ /// Attach the session's table (UDTF) functions so that they appear in
+ /// `information_schema.routines` / `SHOW FUNCTIONS`.
+ pub fn with_table_functions(
+ mut self,
+ table_functions: HashMap<String, Arc<TableFunction>>,
+ ) -> Self {
+ self.config.table_functions = table_functions;
+ self
+ }
}
#[derive(Clone, Debug)]
struct InformationSchemaConfig {
catalog_list: Arc<dyn CatalogProviderList>,
+ table_functions: HashMap<String, Arc<TableFunction>>,
Review Comment:
Thanks @alamb for this, TableFunction lives in datafusion-catalog, which already depends on datafusion-session — putting table_functions() on the Session trait reverses that edge. Scalar/Agg live in datafusion-expr (below session), so no cycle there. Same root as #23348 — I'll open a follow-up PR that moves TableFunction down and deletes with_table_functions().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas merged PR #23438:
URL: https://github.com/apache/datafusion/pull/23438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #23438:
URL: https://github.com/apache/datafusion/pull/23438#issuecomment-4998838553
Thank you @alamb and @xudong963 for review, merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #23438:
URL: https://github.com/apache/datafusion/pull/23438#discussion_r3557043874
##########
datafusion/sql/src/statement.rs:
##########
@@ -2641,7 +2643,7 @@ SELECT DISTINCT
FROM
(
SELECT
- i.specific_name function_name,
+ o.specific_name function_name,
o.data_type return_type,
array_agg(i.parameter_name ORDER BY i.ordinal_position ASC) parameters,
array_agg(i.data_type ORDER BY i.ordinal_position ASC) parameter_types
Review Comment:
With the new LEFT JOIN, functions that have an OUT row but no IN rows (e.g. UDTFs) will produce a single joined row where `i.*` is NULL. `array_agg` includes nulls by default, so `parameters` / `parameter_types` will become `[NULL]` rather than empty/NULL, which is a user-visible regression in `SHOW FUNCTIONS` output.
##########
datafusion/sql/src/statement.rs:
##########
@@ -2631,6 +2631,8 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
"".to_string()
};
+ // Note: use LEFT JOIN from OUT rows to IN rows so functions without
+ // input parameters (notably UDTFs / table functions) still appear.
let query = format!(
Review Comment:
This change adds new user-visible rows for table functions (UDTFs) in `information_schema.routines` / `information_schema.parameters` and affects `SHOW FUNCTIONS` output. There are existing sqllogictests in `datafusion/sqllogictest/test_files/information_schema.slt` that cover routines/parameters and SHOW FUNCTIONS, but none that assert table functions appear. Adding a small assertion for a built-in table function (e.g. `unnest`) would prevent regressions.
##########
datafusion/catalog/src/information_schema.rs:
##########
@@ -301,6 +316,25 @@ impl InformationSchemaConfig {
)
}
}
+
+ // Table functions (UDTFs) don't have scalar signatures; their return
+ // type is always a table, so emit a single row per UDTF with
+ // routine_type = "TABLE" and data_type = "TABLE".
Review Comment:
The comment says `routine_type = "TABLE"`, but the code correctly sets `routine_type` to "FUNCTION" and `function_type` to "TABLE". This is likely to confuse future maintainers reading the information_schema mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #23438:
URL: https://github.com/apache/datafusion/pull/23438#discussion_r3592437393
##########
datafusion/catalog/src/information_schema.rs:
##########
@@ -81,14 +82,28 @@ impl InformationSchemaProvider {
/// Creates a new [`InformationSchemaProvider`] for the provided `catalog_list`
pub fn new(catalog_list: Arc<dyn CatalogProviderList>) -> Self {
Self {
- config: InformationSchemaConfig { catalog_list },
+ config: InformationSchemaConfig {
+ catalog_list,
+ table_functions: HashMap::new(),
+ },
}
}
+
+ /// Attach the session's table (UDTF) functions so that they appear in
+ /// `information_schema.routines` / `SHOW FUNCTIONS`.
+ pub fn with_table_functions(
+ mut self,
+ table_functions: HashMap<String, Arc<TableFunction>>,
+ ) -> Self {
+ self.config.table_functions = table_functions;
+ self
+ }
}
#[derive(Clone, Debug)]
struct InformationSchemaConfig {
catalog_list: Arc<dyn CatalogProviderList>,
+ table_functions: HashMap<String, Arc<TableFunction>>,
Review Comment:
Thanks @alamb for good catch, refreshed the PR description with the concrete cycle explanation.
TableFunction lives in datafusion-catalog, which already depends on datafusion-session — putting table_functions() on the Session trait reverses that edge. Scalar/Agg live in datafusion-expr (below session), so no cycle there. Same root as #23348 — I'll open a follow-up PR that moves TableFunction down and deletes with_table_functions().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support UDTFs in information_schema.routines / SHOW FUNCTIONS [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #23438:
URL: https://github.com/apache/datafusion/pull/23438#discussion_r3587807460
##########
datafusion/catalog/src/information_schema.rs:
##########
@@ -81,14 +82,28 @@ impl InformationSchemaProvider {
/// Creates a new [`InformationSchemaProvider`] for the provided `catalog_list`
pub fn new(catalog_list: Arc<dyn CatalogProviderList>) -> Self {
Self {
- config: InformationSchemaConfig { catalog_list },
+ config: InformationSchemaConfig {
+ catalog_list,
+ table_functions: HashMap::new(),
+ },
}
}
+
+ /// Attach the session's table (UDTF) functions so that they appear in
+ /// `information_schema.routines` / `SHOW FUNCTIONS`.
+ pub fn with_table_functions(
+ mut self,
+ table_functions: HashMap<String, Arc<TableFunction>>,
+ ) -> Self {
+ self.config.table_functions = table_functions;
+ self
+ }
}
#[derive(Clone, Debug)]
struct InformationSchemaConfig {
catalog_list: Arc<dyn CatalogProviderList>,
+ table_functions: HashMap<String, Arc<TableFunction>>,
Review Comment:
I don't understand why we have to treat TableFunctions specially -- the PR descriptio says it is to avoid a crate dependency cycle, but I don't understand why the same thing doesn't apply to Scalar and Aggregate functons
Mabe we just need to move a definition of TableFunction to datafusion-expr 🤔 But maybe that is not possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org