You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2026/03/02 00:56:56 UTC
[PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
lidavidm opened a new pull request, #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031
Closes #3983.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2892805383
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
It's on my list, I just need to get to it!
I'm not sure that metadata is necessarily commonly exposed, or that a driver can necessarily predict the return type (outside of something like DataFusion). JDBC/ODBC seem to assume a fixed list of overloads, for instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#issuecomment-4444583799
There's always a tension when fetching metadata between the desire to minimize round-trips and the cost of fetching or examining things you don't really need. The latter is worsened in situations where looking at metadata involves taking a lock on metadata tables, which means both that you may be blocked longer waiting to get a lock and that you may end up blocking someone else with your own locks.
Our experience with `GetObjects` was that (at least for Snowflake) it was too slow for our purposes.
When in doubt, following the pattern of existing ODBC/JDBC APIs probably makes sense. Even where those APIs are flawed, it's likely that existing systems have done some work to accommodate those patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2891131265
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
Yes, but it's not in the spec yet 🙂
I still think serializing function arguments and results as flatbuffers is pretty gross and potentially not even that useful (the actual output type sometimes depends on actual literal arguments). A driver-specific string or binary representation of a parameter data type and a function that converts that driver-specific string representation (no roundtrip needed) would solve both of those issues without flatbuffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed pull request #4031: feat(format): expose functions/procedures in catalog metadata
URL: https://github.com/apache/arrow-adbc/pull/4031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2870395009
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
https://github.com/apache/arrow-adbc/issues/3793 and https://github.com/apache/arrow-adbc/issues/3996 are related to that, I believe.
I thought about a separate function, but note that even for GetObjects there's a request to add a serialized schema to avoid round trips.
https://github.com/apache/arrow-adbc/issues/1704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2893770333
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
Ah, is the concern mostly about not being human-readable? I agree there...I don't want to go yak shave the JSON issue but maybe it's time...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2893751640
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1989,6 +1993,103 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |-----------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |-----------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |-----------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_examples | list<utf8> | (3) |
+/// | routine_definition | utf8 | (4) |
+/// | routine_definition_language | utf8 | (4) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (5) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (5) |
+/// | routine_parameter_schema | binary | (6) |
+/// | routine_result_schema | binary | (6) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description or help text, along with examples of the
+/// syntax.
+/// 4. The definition (e.g. SQL text used to create a procedure) and the
+/// language of the definition (e.g. SQL, Python)
+/// 5. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 6. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
+///
+/// PARAMETER_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |-----------------------------|-------------------------|----------|
+/// | param_name | utf8 not null | |
+/// | ordinal_position | int32 | (1) |
+/// | remarks | utf8 | (2) |
+/// | param_type | utf8 | (3) |
+/// | xdbc_data_type | int16 | (3) |
+/// | xdbc_type_name | utf8 | (3) |
+/// | xdbc_precision | int32 | (3) |
+/// | xdbc_length | int32 | (3) |
+/// | xdbc_scale | int16 | (3) |
+/// | xdbc_num_prec_radix | int16 | (3) |
+/// | xdbc_nullable | int16 | (3) |
+/// | xdbc_char_octet_length | int32 | (3) |
+/// | xdbc_is_nullable | utf8 | (3) |
+///
Review Comment:
There are a few others here if we're emulating SQLGetTypeInfo:
-
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1989,6 +1993,103 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |-----------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |-----------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |-----------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_examples | list<utf8> | (3) |
+/// | routine_definition | utf8 | (4) |
+/// | routine_definition_language | utf8 | (4) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (5) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (5) |
+/// | routine_parameter_schema | binary | (6) |
+/// | routine_result_schema | binary | (6) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description or help text, along with examples of the
+/// syntax.
+/// 4. The definition (e.g. SQL text used to create a procedure) and the
+/// language of the definition (e.g. SQL, Python)
+/// 5. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 6. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
+///
+/// PARAMETER_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |-----------------------------|-------------------------|----------|
+/// | param_name | utf8 not null | |
+/// | ordinal_position | int32 | (1) |
+/// | remarks | utf8 | (2) |
+/// | param_type | utf8 | (3) |
+/// | xdbc_data_type | int16 | (3) |
+/// | xdbc_type_name | utf8 | (3) |
+/// | xdbc_precision | int32 | (3) |
+/// | xdbc_length | int32 | (3) |
+/// | xdbc_scale | int16 | (3) |
+/// | xdbc_num_prec_radix | int16 | (3) |
+/// | xdbc_nullable | int16 | (3) |
+/// | xdbc_char_octet_length | int32 | (3) |
+/// | xdbc_is_nullable | utf8 | (3) |
+///
Review Comment:
There are a few others here if we're emulating SQLGetTypeInfo:
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2870395908
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
Review Comment:
That's probably something I should investigate and add to the list! I believe some other systems also provide the definition for UDFs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2893816160
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
Also the vendor-specific type is already there as `xdbc_type_name`, unless there's more metadata you're expecting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#issuecomment-3981463903
Some notes:
- There may be more metadata that is not in JDBC/ODBC, but which is reasonably common that we can standardize?
- I've chosen not to model the types of multiple result sets; based on my search, almost nothing actually exposes this metadata (and it may not even be possible to determine the type with static analysis). But we should add a note about this.
- I need to describe the Arrow schema field in more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#issuecomment-4444178103
Is there a particular reason to add a new function here rather than integrating this with the existing GetObjects function to allow it to return tables AND functions/procedures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#issuecomment-4444210223
It's way too heavy to add so much to one function, and we would have to add a bunch more API surface anyways to control whether to return functions/procedures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2870295848
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
Review Comment:
DataFusion and DuckDB both have "description" and SQL examples (useful for generating help pages or inline help). Where would these go?
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
+///
+/// PARAMETER_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | param_name | utf8 not null | |
+/// | ordinal_position | int32 | (1) |
+/// | remarks | utf8 | (2) |
+/// | param_type | utf8 | (3) |
+/// | xdbc_data_type | int16 | (3) |
+/// | xdbc_type_name | utf8 | (3) |
+/// | xdbc_precision | int32 | (3) |
+/// | xdbc_length | int32 | (3) |
+/// | xdbc_scale | int16 | (3) |
+/// | xdbc_num_prec_radix | int16 | (3) |
+/// | xdbc_nullable | int16 | (3) |
+/// | xdbc_char_octet_length | int32 | (3) |
+/// | xdbc_is_nullable | utf8 | (3) |
Review Comment:
It is a good thing LLMs are much better than they were when creating builders/parsers for the last nested GetObjects schema 🙂
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
How about another function(s) instead of invoking flatbuffers and only returning if a flag is set?
```c
AdbcConnectionGetRoutineParameterSchema(struct AdbcConnection* connection, const char* catalog, const char* db_schema, const char* routine_name, struct AdbcSchema** parameter_schema_out, int n_parameter_schema_out, struct AdbcSchema** result_schema_out, int n_result_schema_out, struct AdbcError* error);
```
...or something.
A related question whose answer is often as useful (or more useful) is: given a list of arrow types (and the scalar argument values, if present), what Arrow type would you give me back? This is roughly DataFusion's `ScalarUdfImpl::return_field()`. I have an ABI for this because I have to write a bunch of our UDFs in C++:
https://github.com/apache/sedona-db/blob/8b16f39b8a8806daacf3c09a2f6e1c864fed69e8/c/sedona-extension/src/sedona_extension.h#L148-L151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2893744933
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
+/// 1. A name that uniquely identifies the routine, to disambiguate
+/// overloads.
+/// 2. 'FUNCTION', 'PROCEDURE', or a vendor-specific name (e.g. 'TABLE
+/// FUNCTION').
+/// 3. Vendor-specific description of the routine.
+/// 4. Metadata about the accepted parameters and return values as structured
+/// Arrow data. Only populated if include_columns is set, otherwise null.
+/// 5. Metadata about the accepted parameters and return values as an Arrow
+/// schema, serialized as an IPC message containing a schema Flatbuffers
+/// structure. Only populated if include_arrow_schema is set, otherwise
+/// null.
Review Comment:
> I'm not sure that metadata is necessarily commonly exposed
Yes, this is probably unique to systems where the driver is the database (and decimals have type-level precision/scale). It is roughly what is needed to do syntax checking on SQL expressions although can be approximated by a fixed list of overloads of coarser types.
Maybe if a fully serialized vendor/driver-specific type (e.g., what would appear in CREATE TABLE) were available in the parameter schema and we had a function to convert that to an arrow type? (The function that can do the reverse of that would be #3793 I think)
Or you can ignore me and funnel Arrow IPC via the Arrow C Data interface 🙂 (I just think it's a shame we never agreed on a string representation of a data type that would make this cleaner)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#discussion_r2888298211
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1858,6 +1858,96 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Get a hierarchical view of all functions and procedures.
+///
+/// The result is an Arrow dataset with the following schema:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | catalog_name | utf8 |
+/// | catalog_db_schemas | list<DB_SCHEMA_SCHEMA> |
+///
+/// DB_SCHEMA_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type |
+/// |--------------------------|-------------------------|
+/// | db_schema_name | utf8 |
+/// | db_schema_routines | list<ROUTINE_SCHEMA> |
+///
+/// ROUTINE_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|-------------------------|----------|
+/// | routine_name | utf8 not null | |
+/// | routine_specific_name | utf8 not null | (1) |
+/// | routine_type | utf8 not null | (2) |
+/// | routine_remarks | utf8 | (3) |
+/// | routine_parameters | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_result | list<PARAMETER_SCHEMA> | (4) |
+/// | routine_parameter_schema | binary | (5) |
+/// | routine_result_schema | binary | (5) |
+///
Review Comment:
I updated this: `remarks` can contain the main docstring, `examples` can contain a list of examples, and now there are two fields for the definition (e.g. for a UDF) and the definition language.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): expose functions/procedures in catalog metadata [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4031:
URL: https://github.com/apache/arrow-adbc/pull/4031#issuecomment-5435595910
I folded this into https://github.com/apache/arrow-adbc/pull/4694.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org