You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zeroshade (via GitHub)" <gi...@apache.org> on 2024/07/18 19:46:02 UTC
[PR] feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest [arrow-adbc]
zeroshade opened a new pull request, #2025:
URL: https://github.com/apache/arrow-adbc/pull/2025
Closes #2005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #2025:
URL: https://github.com/apache/arrow-adbc/pull/2025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #2025:
URL: https://github.com/apache/arrow-adbc/pull/2025#discussion_r1683603992
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -45,7 +45,7 @@ import (
const (
bindStageName = "ADBC$BIND"
- createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE)"
+ createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE USE_VECTORIZED_SCANNER=TRUE REPLACE_INVALID_CHARACTERS = TRUE)"
Review Comment:
Is it necessary to set REPLACE_INVALID_CHARACTERS? I guess Snowflake recommends it "as a general rule of thumb" but I wonder why...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #2025:
URL: https://github.com/apache/arrow-adbc/pull/2025#issuecomment-2245393911
> Does this have any effect on performance today, or will we need to wait until Snowflake releases the follow-up changes?
It depends on the configuration of the warehouse, some configurations (encryption, security settings, etc.) are not yet supported, while others are. So if your configuration is supported, you'll get benefits immediately while other configurations will need to wait until snowflake makes some more changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #2025:
URL: https://github.com/apache/arrow-adbc/pull/2025#discussion_r1688158952
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -45,7 +45,7 @@ import (
const (
bindStageName = "ADBC$BIND"
- createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE)"
+ createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE USE_VECTORIZED_SCANNER=TRUE REPLACE_INVALID_CHARACTERS = TRUE)"
Review Comment:
I don't know, maybe just to reduce failures caused by the characters? Not sure myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest [arrow-adbc]
Posted by "sfc-gh-xhuang (via GitHub)" <gi...@apache.org>.
sfc-gh-xhuang commented on code in PR #2025:
URL: https://github.com/apache/arrow-adbc/pull/2025#discussion_r1697645314
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -45,7 +45,7 @@ import (
const (
bindStageName = "ADBC$BIND"
- createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE)"
+ createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE USE_VECTORIZED_SCANNER=TRUE REPLACE_INVALID_CHARACTERS = TRUE)"
Review Comment:
It's a best practice as invalid UTF8 can cause query failures or unexpected results:
https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-1013-1014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org