You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "msrathore-db (via GitHub)" <gi...@apache.org> on 2025/10/04 10:03:01 UTC
[PR] feat(csharp/src/Drivers/Databricks): Used connection param of batchSize for cloudFetch. [arrow-adbc]
msrathore-db opened a new pull request, #3518:
URL: https://github.com/apache/arrow-adbc/pull/3518
Used connection param of batchSize for cloudFetch. Changed the default value of the param to 2 million instead of 50k.
[PECO-2732](https://databricks.atlassian.net/browse/PECO-2732)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Used connection param of batchSize for cloudFetch. [arrow-adbc]
Posted by "eric-wang-1990 (via GitHub)" <gi...@apache.org>.
eric-wang-1990 commented on code in PR #3518:
URL: https://github.com/apache/arrow-adbc/pull/3518#discussion_r2403995940
##########
csharp/src/Drivers/Databricks/DatabricksStatement.cs:
##########
@@ -36,13 +36,16 @@ namespace Apache.Arrow.Adbc.Drivers.Databricks
/// </summary>
internal class DatabricksStatement : SparkStatement, IHiveServer2Statement
{
+ private const long DatabricksBatchSizeDefault = 2000000;
private bool useCloudFetch;
private bool canDecompressLz4;
private long maxBytesPerFile;
private bool enableMultipleCatalogSupport;
private bool enablePKFK;
private bool runAsyncInThrift;
+ public new long BatchSize { get; private set; } = DatabricksBatchSizeDefault;
Review Comment:
Searching from web, this new keyword is hiding the base class impl, not truly overriding it. Which means if the object is referenced by base class type it will still call base class function.
Can we change the base to virtual so we can call this override?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Used connection param of batchSize for cloudFetch. [arrow-adbc]
Posted by "eric-wang-1990 (via GitHub)" <gi...@apache.org>.
eric-wang-1990 commented on code in PR #3518:
URL: https://github.com/apache/arrow-adbc/pull/3518#discussion_r2403996524
##########
csharp/src/Drivers/Databricks/DatabricksStatement.cs:
##########
@@ -36,13 +36,16 @@ namespace Apache.Arrow.Adbc.Drivers.Databricks
/// </summary>
internal class DatabricksStatement : SparkStatement, IHiveServer2Statement
{
+ private const long DatabricksBatchSizeDefault = 2000000;
Review Comment:
Let's add a comment of explaining why this is set to 2 million.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Used connection param of batchSize for cloudFetch [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #3518:
URL: https://github.com/apache/arrow-adbc/pull/3518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org