You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/06 21:13:10 UTC
[GitHub] [arrow-adbc] lidavidm opened a new issue, #320: [Format] Add arbitrarily nested (metadata) schemas
lidavidm opened a new issue, #320:
URL: https://github.com/apache/arrow-adbc/issues/320
ADBC's model of database metadata is limited to a catalog-schema-table hierarchy; some systems allow unlimited nesting. (See #46.)
Proposal:
- Nested schemas are allowed in the result of ConnectionGetObjects, but will be flattened in the ADBC representation. (This is somewhat unavoidable since you can’t model arbitrarily recursively nested data in Arrow.)
- Nested schemas are flattened by joining the names with a backend-specific delimiter.
- Add a new key for ConnectionGetInfo to get the delimiter value.
For example: a table like `postgres1.dbname.schemaname.table` (which has 4 layers of hierarchy rather than 3) would be represented as: catalog = `postgres1`, db_schema = `dbname.schemaname`, table = `table`.
This proposal applies across languages. See #317 for how to manage backwards compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Add arbitrarily nested (metadata) schemas [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #320: [Format] Add arbitrarily nested (metadata) schemas
URL: https://github.com/apache/arrow-adbc/issues/320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] WillAyd commented on issue #320: [Format] Add arbitrarily nested (metadata) schemas
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-1587740770
I don't know that I've personally worked with a database that allows for arbitrarily nested schemas but would be curious to know how client tools handle this via ODBC/JDBC today. I'm guessing that a nested schema would be represented as `schema_parent.schema_child` in most if not all cases, so I think we could fit that same pattern into the GetObjects design we have today without changing much.
You would lose a "clean" understanding of the schema hierarchy, but AFAIK that is not a SQL standard anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Add arbitrarily nested (metadata) schemas [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-2954343074
This has been bugging me for a while now but I thought of a different approach: we could keep the existing schema for the schema portion of GetObjects and add a new column for "full name", which is `list<string>`. Clients that don't recognize this but connect to a system supporting this may see duplicate schema names (since schema name would remain the same to avoid questions about how to escape things), which is perhaps not ideal. There's also the question of how filters apply to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] lidavidm commented on issue #320: [Format] Add arbitrarily nested (metadata) schemas
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-1553321588
There's some ambiguity here; what should we do if a schema name (for whatever reason) contains the delimiter? Normally the driver would be returning 'unescaped' names, but now there's nothing we can do here.
A different solution might be to make the name columns list<utf8>. But this complicates the common case (unfortunately) and would be a hard breaking change (even if not on the API level, no current client would be able to make use of the new schema).
It would be good to combine this with #621 for a future 'GetObjects2'-style function. (And possibly, break up the nested schema.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Add arbitrarily nested (metadata) schemas [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-5041055132
I think we can centralize discussion in #46.
I don't think trying to flatten into a string with delimiters is viable for the reasons discussed above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] lidavidm commented on issue #320: [Format] Add arbitrarily nested (metadata) schemas
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-1587730650
And hmm, what do JDBC/ODBC do here? JDBC also assumes a fixed catalog/schema/table hierarchy. So what happens for Dremio, etc.?
Dremio doesn't even implement getDbSchemas (IIRC) despite having designed the spec, so not sure if we can use it as a real life example. What does the Trino JDBC driver do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] lidavidm commented on issue #320: [Format] Add arbitrarily nested (metadata) schemas
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-1587723603
It also significantly complicates the filters in the API; again, do you supply the escaped name? Or all functions have to take a list, which is a bit painful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-adbc] lidavidm commented on issue #320: [Format] Add arbitrarily nested (metadata) schemas
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #320:
URL: https://github.com/apache/arrow-adbc/issues/320#issuecomment-1587745806
Yup, I'm just curious about a few things like
- Is the separator always `.`?
- Do these systems allow escaping names that use the separator? If so, how do you even write the name anymore? A schema written in SQL as `"schema.1"` would be represented in code as just the string `schema.1` (i.e. you don't work with/store escaped names, but unescaped names), but then what about `"schema.1"."subschema"`?
etc.
And I wonder if these systems even expose this info in their metadata in the first place, given the main use case appears to be pulling in data from external systems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org