You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2026/07/20 06:55:15 UTC
[PR] feat(format): add generic metadata API [WIP] [arrow-adbc]
lidavidm opened a new pull request, #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544
- Add a generic API for fetching metadata, modeled on ADO.NET's GetSchema
- [ ] Define the standard metadata collections
- [ ] Revert the current revisions to GetObjects in favor of not changing it
Related:
- #3997: define a way to escape search pattern arguments
- #1704 (comment): define a way to provide pagination tokens
For consideration:
- [ ] Should filters be (optionally) named arguments? (Pointer to struct) The justification: we could more easily implement/define "standard" arguments to handle things like the pagination token above without having to replicate it into every single definition
- [ ] Should filters be typed? (Pointer to struct containing union)
- [ ] Should filters be replaced by just leveraging SetOption(Int,...)? (This would obviate the former two points but is rather clunky, and precludes a hypothetical driver that could handle these operations concurrently)
Closes #4400.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5279245532
I think I'm leaning towards having all options/filters be set by SetOption. This is perhaps inconvenient for C/C++ users, but language-level bindings can present higher level APIs, and allows us to express better type safety. This would also be consistent with my suggestion in #4317.
Either way, the current declaration needs to clarify the lifetime of the filter argument anyways (the driver should copy arguments as it is not allowed to assume filters will be valid during the returned record reader's lifetime).
I thought about having a SetOptionStringList. This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts, and I would rather avoid trying to encode strings into a single string (via e.g. JSON). But maybe we can embrace Curt's suggestions and lean on Parquet Variant for encoding these sorts of complex-type arguments. That said, I fear I'm reinventing COM or some sort of intraprocess RPC mechanism...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed pull request #4544: feat(format): add generic metadata API
URL: https://github.com/apache/arrow-adbc/pull/4544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5312449256
Drafted that at https://github.com/apache/arrow-adbc/pull/4694.
I don't love that API either but I think it is more consistent than this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [WIP] [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5019792251
At the risk of overcomplicating things: similar to #3623, it might be nice to have a way to request that certain extra fields be included/omitted, e.g. table properties (#3995), where it may be efficient to fetch the data at the same time as the "standard" fields, but where some (many) clients also may not want the field. Maybe it could optionally take an Arrow schema as input for that.
Similarly, maybe the application wants to opt in to run-length-encoding certain response columns to save memory. (Or is that not really a concern so long as things are properly streamed/paginated?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5290164924
Also, I think encoding filter arguments is good _enough_. At least with current APIs, essentially all filter arguments are strings anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#discussion_r3780300834
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2122,6 +2134,360 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Fetch (catalog) metadata from the database.
+///
+/// The metadata to fetch is defined by the `collection` parameter. The result
+/// is an Arrow dataset with a schema defined by the collection. For example,
+/// a client may request a list of tables in the database, or a list of
+/// supported data types. Drivers may implement collections beyond those
+/// defined by ADBC, but must use a vendor-specific prefix
+/// (e.g. `postgresql.`) to avoid conflicts with future standardized
+/// collections. Drivers must not use the `adbc.` prefix.
+///
+/// The result may be filtered by `filters`, which is an array of (nullable)
+/// strings. `num_filters` must be set to the number of filter arguments
+/// passed.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to
+/// NULL and blank string) that defines the available collections. See
+/// ADBC_METADATA_COLLECTION_META.
+///
+/// Drivers may add more fields at the end of standard schemas to reflect
+/// vendor-specific metadata. Applications must access these using an offset
+/// from the end of the schema and cannot assume that the index of the field
+/// will remain stable. Drivers must add the fields at the end should prefix
+/// field names with the vendor/driver name to differentiate them
+/// (e.g. 'POSTGRESQL:owner').
+///
+/// Similarly, future standard revisions may add more fields to existing
+/// standard schemas. Applications must not assume the number of fields is
+/// fixed.
+///
+/// This AdbcConnection must outlive the returned ArrowArrayStream.
+///
+/// \param[in] connection The database connection.
+/// \param[in] collection The collection to fetch.
+/// \param[out] out The result set.
Review Comment:
The filters params are missing from the docs here
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2122,6 +2134,360 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Fetch (catalog) metadata from the database.
+///
+/// The metadata to fetch is defined by the `collection` parameter. The result
+/// is an Arrow dataset with a schema defined by the collection. For example,
+/// a client may request a list of tables in the database, or a list of
+/// supported data types. Drivers may implement collections beyond those
+/// defined by ADBC, but must use a vendor-specific prefix
+/// (e.g. `postgresql.`) to avoid conflicts with future standardized
+/// collections. Drivers must not use the `adbc.` prefix.
+///
+/// The result may be filtered by `filters`, which is an array of (nullable)
+/// strings. `num_filters` must be set to the number of filter arguments
+/// passed.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to
+/// NULL and blank string) that defines the available collections. See
+/// ADBC_METADATA_COLLECTION_META.
+///
+/// Drivers may add more fields at the end of standard schemas to reflect
+/// vendor-specific metadata. Applications must access these using an offset
+/// from the end of the schema and cannot assume that the index of the field
+/// will remain stable. Drivers must add the fields at the end should prefix
+/// field names with the vendor/driver name to differentiate them
+/// (e.g. 'POSTGRESQL:owner').
+///
+/// Similarly, future standard revisions may add more fields to existing
+/// standard schemas. Applications must not assume the number of fields is
+/// fixed.
+///
+/// This AdbcConnection must outlive the returned ArrowArrayStream.
+///
+/// \param[in] connection The database connection.
+/// \param[in] collection The collection to fetch.
+/// \param[out] out The result set.
+/// \param[out] error Error details, if an error occurs.
+/// \since ADBC API revision 1.2.0
+ADBC_EXPORT
+AdbcStatusCode AdbcConnectionGetMetadataCollection(
+ struct AdbcConnection* connection, const char* collection, size_t num_filters,
+ const char** filters, struct ArrowArrayStream* out, struct AdbcError* error);
+
+/// \brief The "meta" collection returns the available metadata collections.
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | collection_name | utf8 not null | |
+/// | collection_description | utf8 | |
+/// | collection_schema | extension<arrow.schema_json> | |
+/// | collection_filters | list<FILTER_SCHEMA> | |
+///
+/// FILTER_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | filter_description | utf8 | |
+/// | required | bool not null | |
+#define ADBC_METADATA_COLLECTION_META "meta"
+
+/// \brief The "catalogs" collection returns the catalogs defined in the
+/// database.
+///
+/// Some systems may not have the concept of catalogs, in which case this
+/// collection should contain a single entry with an empty, non-null name.
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | catalog_name | utf8 | |
+/// | catalog_remarks | utf8 | (1) |
+///
+/// (1) A description of the catalog.
+///
+/// Filters:
+/// 1. The catalog name to filter by. May be a search pattern.
+#define ADBC_METADATA_COLLECTION_CATALOGS "catalogs"
+
+/// \brief The "schemas" collection returns the schemas defined in the
+/// database.
+///
+/// Some systems may not have the concept of schemas, in which case this
+/// collection should contain a single entry per catalog with an empty,
+/// non-null name.
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | catalog_name | utf8 | (R) |
+/// | db_schema_name | utf8 | |
+/// | db_schema_remarks | utf8 | (1) |
+///
+/// (R) This field is run-length encoded by default; it can be disabled via
Review Comment:
Why REE instead of dictionary encoded (which is more widely supported among arrow implementations)?
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2122,6 +2134,360 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Fetch (catalog) metadata from the database.
+///
+/// The metadata to fetch is defined by the `collection` parameter. The result
+/// is an Arrow dataset with a schema defined by the collection. For example,
+/// a client may request a list of tables in the database, or a list of
+/// supported data types. Drivers may implement collections beyond those
+/// defined by ADBC, but must use a vendor-specific prefix
+/// (e.g. `postgresql.`) to avoid conflicts with future standardized
+/// collections. Drivers must not use the `adbc.` prefix.
+///
+/// The result may be filtered by `filters`, which is an array of (nullable)
+/// strings. `num_filters` must be set to the number of filter arguments
+/// passed.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to
+/// NULL and blank string) that defines the available collections. See
+/// ADBC_METADATA_COLLECTION_META.
+///
+/// Drivers may add more fields at the end of standard schemas to reflect
+/// vendor-specific metadata. Applications must access these using an offset
+/// from the end of the schema and cannot assume that the index of the field
+/// will remain stable. Drivers must add the fields at the end should prefix
+/// field names with the vendor/driver name to differentiate them
+/// (e.g. 'POSTGRESQL:owner').
+///
+/// Similarly, future standard revisions may add more fields to existing
+/// standard schemas. Applications must not assume the number of fields is
+/// fixed.
+///
+/// This AdbcConnection must outlive the returned ArrowArrayStream.
+///
+/// \param[in] connection The database connection.
+/// \param[in] collection The collection to fetch.
+/// \param[out] out The result set.
+/// \param[out] error Error details, if an error occurs.
+/// \since ADBC API revision 1.2.0
+ADBC_EXPORT
+AdbcStatusCode AdbcConnectionGetMetadataCollection(
+ struct AdbcConnection* connection, const char* collection, size_t num_filters,
+ const char** filters, struct ArrowArrayStream* out, struct AdbcError* error);
Review Comment:
Did I miss it or does the docs not define the semantics of the filters in terms of prefix, a LIKE pattern, regex, etc.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5296476517
> do you mean that it's a flat list of alternating [key, value, key, value, key, value]
Yes, like `auto options = {"METADATA_ENCODING=geoarrow", "FOOFY2=something else", nullptr};`. No need to do anything if the existing filter specification works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [WIP] [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5019608401
Also CC @iconara
I've also been thinking about your request for stateless pagination, and I think it's reasonable enough to define a way to get/pass a pagination token. I think there are enough systems that could use it: Athena, BigQuery, Databricks, Snowflake, Iceberg REST catalog, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [WIP] [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5019631672
And also CC @mullinsms, who kicked all this off by listing all the types of metadata we didn't support :slightly_smiling_face:
In particular, Curt's suggestion means we can support database-specific metadata (though, I'm still of the opinion that something truly database-specific is probably better handled by the application; if the application has a very specific metadata query it wants to issue I'm not sure how useful it is to build it into the driver).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#discussion_r3780364274
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2122,6 +2134,360 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Fetch (catalog) metadata from the database.
+///
+/// The metadata to fetch is defined by the `collection` parameter. The result
+/// is an Arrow dataset with a schema defined by the collection. For example,
+/// a client may request a list of tables in the database, or a list of
+/// supported data types. Drivers may implement collections beyond those
+/// defined by ADBC, but must use a vendor-specific prefix
+/// (e.g. `postgresql.`) to avoid conflicts with future standardized
+/// collections. Drivers must not use the `adbc.` prefix.
+///
+/// The result may be filtered by `filters`, which is an array of (nullable)
+/// strings. `num_filters` must be set to the number of filter arguments
+/// passed.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to
+/// NULL and blank string) that defines the available collections. See
+/// ADBC_METADATA_COLLECTION_META.
+///
+/// Drivers may add more fields at the end of standard schemas to reflect
+/// vendor-specific metadata. Applications must access these using an offset
+/// from the end of the schema and cannot assume that the index of the field
+/// will remain stable. Drivers must add the fields at the end should prefix
+/// field names with the vendor/driver name to differentiate them
+/// (e.g. 'POSTGRESQL:owner').
+///
+/// Similarly, future standard revisions may add more fields to existing
+/// standard schemas. Applications must not assume the number of fields is
+/// fixed.
+///
+/// This AdbcConnection must outlive the returned ArrowArrayStream.
+///
+/// \param[in] connection The database connection.
+/// \param[in] collection The collection to fetch.
+/// \param[out] out The result set.
+/// \param[out] error Error details, if an error occurs.
+/// \since ADBC API revision 1.2.0
+ADBC_EXPORT
+AdbcStatusCode AdbcConnectionGetMetadataCollection(
+ struct AdbcConnection* connection, const char* collection, size_t num_filters,
+ const char** filters, struct ArrowArrayStream* out, struct AdbcError* error);
Review Comment:
Er, the point of this is that each collection can define its own filters...see the collection definitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#discussion_r3781529120
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -2122,6 +2134,360 @@ AdbcStatusCode AdbcConnectionGetObjects(struct AdbcConnection* connection, int d
struct ArrowArrayStream* out,
struct AdbcError* error);
+/// \brief Fetch (catalog) metadata from the database.
+///
+/// The metadata to fetch is defined by the `collection` parameter. The result
+/// is an Arrow dataset with a schema defined by the collection. For example,
+/// a client may request a list of tables in the database, or a list of
+/// supported data types. Drivers may implement collections beyond those
+/// defined by ADBC, but must use a vendor-specific prefix
+/// (e.g. `postgresql.`) to avoid conflicts with future standardized
+/// collections. Drivers must not use the `adbc.` prefix.
+///
+/// The result may be filtered by `filters`, which is an array of (nullable)
+/// strings. `num_filters` must be set to the number of filter arguments
+/// passed.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to
+/// NULL and blank string) that defines the available collections. See
+/// ADBC_METADATA_COLLECTION_META.
+///
+/// Drivers may add more fields at the end of standard schemas to reflect
+/// vendor-specific metadata. Applications must access these using an offset
+/// from the end of the schema and cannot assume that the index of the field
+/// will remain stable. Drivers must add the fields at the end should prefix
+/// field names with the vendor/driver name to differentiate them
+/// (e.g. 'POSTGRESQL:owner').
+///
+/// Similarly, future standard revisions may add more fields to existing
+/// standard schemas. Applications must not assume the number of fields is
+/// fixed.
+///
+/// This AdbcConnection must outlive the returned ArrowArrayStream.
+///
+/// \param[in] connection The database connection.
+/// \param[in] collection The collection to fetch.
+/// \param[out] out The result set.
+/// \param[out] error Error details, if an error occurs.
+/// \since ADBC API revision 1.2.0
+ADBC_EXPORT
+AdbcStatusCode AdbcConnectionGetMetadataCollection(
+ struct AdbcConnection* connection, const char* collection, size_t num_filters,
+ const char** filters, struct ArrowArrayStream* out, struct AdbcError* error);
+
+/// \brief The "meta" collection returns the available metadata collections.
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | collection_name | utf8 not null | |
+/// | collection_description | utf8 | |
+/// | collection_schema | extension<arrow.schema_json> | |
+/// | collection_filters | list<FILTER_SCHEMA> | |
+///
+/// FILTER_SCHEMA is a Struct with fields:
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | filter_description | utf8 | |
+/// | required | bool not null | |
+#define ADBC_METADATA_COLLECTION_META "meta"
+
+/// \brief The "catalogs" collection returns the catalogs defined in the
+/// database.
+///
+/// Some systems may not have the concept of catalogs, in which case this
+/// collection should contain a single entry with an empty, non-null name.
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | catalog_name | utf8 | |
+/// | catalog_remarks | utf8 | (1) |
+///
+/// (1) A description of the catalog.
+///
+/// Filters:
+/// 1. The catalog name to filter by. May be a search pattern.
+#define ADBC_METADATA_COLLECTION_CATALOGS "catalogs"
+
+/// \brief The "schemas" collection returns the schemas defined in the
+/// database.
+///
+/// Some systems may not have the concept of schemas, in which case this
+/// collection should contain a single entry per catalog with an empty,
+/// non-null name.
+///
+/// | Field Name | Field Type | Comments |
+/// |--------------------------|------------------------------|----------|
+/// | catalog_name | utf8 | (R) |
+/// | db_schema_name | utf8 | |
+/// | db_schema_remarks | utf8 | (1) |
+///
+/// (R) This field is run-length encoded by default; it can be disabled via
Review Comment:
I suppose I don't like the potential complexity around having to handle replacement/delta dictionaries (which I think are also not quite fully supported in all cases), and the statefulness that dictionaries bring (REE data is always fully decodable from a single batch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5310929293
Another thought: what if we just place this as a statement-level function? Even if it's not associated with a statement per se this lets us get access to existing functions like RequestSchema and lets us scope SetOption without "affecting" the entire connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5290153012
> > I think I'm leaning towards having all options/filters be set by SetOption.
>
> This is kind of ugly but I see what you're getting at. GDAL handles this with `const char** options` where options is NULL terminated list of `KEY=arbitrary value`, with the downside that everything is serialized. If this is a get/set option on a connection is seems more problematic (on a statement it's more clear that options have a specific scope).
I agree that the scoping of the option feels a bit awkward. @paleolimbot do you mean that it's a flat list of alternating `[key, value, key, value, key, value]`? That could work well enough.
>
> > This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts
>
> It's a hack, but some REST APIs use the unit separator (`\x1f`) or have a user-specified separator and just have "schemas" with potentially multiple parts.
I suppose the separator works but I dislike this sort of in-band encoding...that said we could also figure out some convention to encode multiple variadic arguments into a parameter list (e.g. terminating a variadic argument with a NULL element)
>
> I'm guessing you don't want to go this direction, but I can't help but notice there's a large amount of complexity associated with stuffing these concepts into Arrow arrays that are highly nested and very difficult to parse. I'll throw out that you could do something like
>
> ```c
> struct AdbcCatalogNode {
> AdbcStatus (*get_property)(struct AdbcCatalogNode* self, const char* what, struct ArrowSchema* out_schema, struct ArrowArray* out_array, AdbcError* err);
> AdbcStatus (*get_child)(struct AdbcCatalogNode* self, const char* what, AdbcError* err);
> void* private_data;
> void* private_driver;
> }
>
> AdbcStatus AdbcConnectionGetCatalogs(struct AdbcConnection* connection, const char** options, struct AdbcCatalogNode* out, AdbcError* err);
> ```
>
> Not perfect, but maybe lets some of this complexity get pushed onto the driver instead of on the consumer since drivers might have abstractions for some of this already.
The new collection definitions don't nest the data anymore, to hopefully avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5322292595
I'll close this in favor of the reworked proposal at https://github.com/apache/arrow-adbc/pull/4694.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(format): add generic metadata API [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #4544:
URL: https://github.com/apache/arrow-adbc/pull/4544#issuecomment-5285528119
> I think I'm leaning towards having all options/filters be set by SetOption.
This is kind of ugly but I see what you're getting at. GDAL handles this with `const char** options` where options is NULL terminated list of `KEY=arbitrary value`, with the downside that everything is serialized. If this is a get/set option on a connection is seems more problematic (on a statement it's more clear that options have a specific scope).
> This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts
It's a hack, but some REST APIs use the unit separator (`\x1f`) or have a user-specified separator and just have "schemas" with potentially multiple parts.
I'm guessing you don't want to go this direction, but I can't help but notice there's a large amount of complexity associated with stuffing these concepts into Arrow arrays that are highly nested and very difficult to parse. I'll throw out that you could do something like
```c
struct AdbcCatalogNode {
AdbcStatus (*get_property)(struct AdbcCatalogNode* self, const char* what, struct ArrowSchema* out_schema, struct ArrowArray* out_array, AdbcError* err);
AdbcStatus (*get_child)(struct AdbcCatalogNode* self, const char* what, AdbcError* err);
void* private_data;
void* private_driver;
}
AdbcStatus AdbcConnectionGetCatalogs(struct AdbcConnection* connection, const char** options, struct AdbcCatalogNode* out, AdbcError* err);
```
Not perfect, but maybe lets some of this complexity get pushed onto the driver instead of on the consumer since drivers might have abstractions for some of this already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org