You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "davidhcoe (via GitHub)" <gi...@apache.org> on 2023/10/25 18:37:17 UTC
[I] go/adbc/driver/snowflake: support for double quoted identifiers [arrow-adbc]
davidhcoe opened a new issue, #1225:
URL: https://github.com/apache/arrow-adbc/issues/1225
In recent testing, databases that contain the `-` character cause errors in the GetInfo calls because of the way the SQL is generated inside of the driver. These database names should be double quoted per https://docs.snowflake.com/en/sql-reference/identifiers-syntax#double-quoted-identifiers but are returned as is in the https://github.com/apache/arrow-adbc/blob/aafb99601825aaed054f0772f952cd14004a1c04/go/adbc/driver/snowflake/connection.go#L278 call.
Additionally, the cursor-based call takes a long time to perform for accounts with a large number of databases. This has been tested to take 12 seconds to over 1 minute to perform this query that typically takes less than a second on ODBC to perform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] go/adbc/driver/snowflake: support for double quoted identifiers [arrow-adbc]
Posted by "ryan-syed (via GitHub)" <gi...@apache.org>.
ryan-syed commented on issue #1225:
URL: https://github.com/apache/arrow-adbc/issues/1225#issuecomment-1803034018
I have been looking into this issue and here is my analysis:
### Time taken by GetObjects initially for my setup with the catalog name and schema name provided:
- Catalogs: ~ (703 ms - 1s)
- DbSchemas: ~ (4 - 6)s
- Tables: ~(8.5 - 10.6)s
- All/Columns: ~12s
### I have replaced the cursor implementation with static calls and the time has reduced to roughly:
- Catalogs ~1s
- DbSchemas ~ 1.2s
- Tables ~ 1.8s
- All/Columns ~ 3s (Here the cursor implementation can't be removed as `SELECT ... FROM <DATABASE_NAME>.INFORMATION_SCHEMA.COLUMNS` requires us to iterate over the databases. However, when the catalog name is known we can filter early)
My understanding of the current implementation is that the cursor first gets a list of databases and then finds all shares with the database name as a pattern. If the share count is greater than zero, it then checks the first row to see if the share isn't associated with a database name. If there isn't a database associated with a share it then skips the database for further query. There are a few issues with this approach as far as I understand:
- The redundant check of _skipping shared dbs with no data and unaccessible_ is performed every time for DbSchemas, Tables and Columns. Even if it is required it can be done once and reused in all further queries
- It checks for all databases even if a database/catalog is provided in the filter.
- If a share isn't associated with a database then `SELECT DATABASE_NAME FROM INFORMATION_SCHEMA.DATABASES` call will not list it and therefore it doesn't seem necessary to call `SHOW SHARES LIKE '%database_name%'` to get a list and check if a DB isn't created for it.
Therefore, we may be fine with just static calls and avoid the cursor implementation altogether for the schemas and tables. If the _**dynamic call of skipping shared DBs with no data**_ is required and I missed something, then it can definitely be reduced to one call instead of three.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] go/adbc/driver/snowflake: support for double quoted identifiers [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #1225:
URL: https://github.com/apache/arrow-adbc/issues/1225#issuecomment-1809061854
I don't think it necessarily matters whether it's a dynamic or static call (for correctness) as long as it is properly skipping DBs that have no data and are inaccessible. I'll take a look at the PR / branch you put up and confirm that it properly skips inaccessible DBs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] go/adbc/driver/snowflake: support for double quoted identifiers [arrow-adbc]
Posted by "ryan-syed (via GitHub)" <gi...@apache.org>.
ryan-syed commented on issue #1225:
URL: https://github.com/apache/arrow-adbc/issues/1225#issuecomment-1815496070
Created a PR: https://github.com/apache/arrow-adbc/pull/1285 for the performance improvement of `GetObjects`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] go/adbc/driver/snowflake: support for double quoted identifiers [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #1225: go/adbc/driver/snowflake: support for double quoted identifiers
URL: https://github.com/apache/arrow-adbc/issues/1225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org