You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2025/10/10 19:34:16 UTC
[PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
CurtHagenlocher opened a new pull request, #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555
The vectorized scanner seems to be causing performance issues under some circumstances. Add an option to disable it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555#discussion_r2421879502
##########
go/adbc/driver/snowflake/statement.go:
##########
@@ -46,6 +46,7 @@ const (
OptionStatementIngestTargetFileSize = "adbc.snowflake.statement.ingest_target_file_size"
OptionStatementIngestCompressionCodec = "adbc.snowflake.statement.ingest_compression_codec" // TODO(GH-1473): Implement option
OptionStatementIngestCompressionLevel = "adbc.snowflake.statement.ingest_compression_level" // TODO(GH-1473): Implement option
+ OptionStatementVectorizedScanner = "adbc.snowflake.statement.use_vectorized_scanner"
Review Comment:
Hmm... should probably reference ingestion. Maybe `ingest_use_vectorized_scanner`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555#discussion_r2426551645
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -171,6 +179,7 @@ func (st *statement) ingestRecord(ctx context.Context) (nrows int64, err error)
})
// Create a temporary stage, we can't start uploading until it has been created
+ createTemporaryStageStmt := fmt.Sprintf(createTemporaryStageTmpl, "")
Review Comment:
Added this for consistency and compatibility, though I'm skeptical that there's any real-world value ;).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555#discussion_r2425072216
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -171,6 +179,7 @@ func (st *statement) ingestRecord(ctx context.Context) (nrows int64, err error)
})
// Create a temporary stage, we can't start uploading until it has been created
+ createTemporaryStageStmt := fmt.Sprintf(createTemporaryStageTmpl, "")
Review Comment:
My read of the code is that this only ever binds a single record so I don't know why you'd ever need the vectorized scanner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555#discussion_r2425017830
##########
go/adbc/driver/snowflake/statement.go:
##########
@@ -46,6 +46,7 @@ const (
OptionStatementIngestTargetFileSize = "adbc.snowflake.statement.ingest_target_file_size"
OptionStatementIngestCompressionCodec = "adbc.snowflake.statement.ingest_compression_codec" // TODO(GH-1473): Implement option
OptionStatementIngestCompressionLevel = "adbc.snowflake.statement.ingest_compression_level" // TODO(GH-1473): Implement option
+ OptionStatementVectorizedScanner = "adbc.snowflake.statement.use_vectorized_scanner"
Review Comment:
Sounds reasonable to me.
(Also not quite an issue for here but the `adbc.` prefix is probably redundant)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555#discussion_r2425018767
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -171,6 +179,7 @@ func (st *statement) ingestRecord(ctx context.Context) (nrows int64, err error)
})
// Create a temporary stage, we can't start uploading until it has been created
+ createTemporaryStageStmt := fmt.Sprintf(createTemporaryStageTmpl, "")
Review Comment:
Why don't we handle the option here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555#discussion_r2425073675
##########
go/adbc/driver/snowflake/bulk_ingestion.go:
##########
@@ -171,6 +179,7 @@ func (st *statement) ingestRecord(ctx context.Context) (nrows int64, err error)
})
// Create a temporary stage, we can't start uploading until it has been created
+ createTemporaryStageStmt := fmt.Sprintf(createTemporaryStageTmpl, "")
Review Comment:
It could bind a single very large record :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(go/adbc/driver/snowflake): Add option to disable vectorized scanner [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #3555:
URL: https://github.com/apache/arrow-adbc/pull/3555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org