You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "eric-wang-1990 (via GitHub)" <gi...@apache.org> on 2025/09/26 20:27:45 UTC
[PR] improve direct query perf [arrow-adbc]
eric-wang-1990 opened a new pull request, #3489:
URL: https://github.com/apache/arrow-adbc/pull/3489
The directResults field control how many rows/bytes can be returned in one arrow batch.
Before this change, due to a bug for databricks it is calling base class SparkConnection, which has maxRows=1000, which is too small.
ODBC can get all results in a single ExecuteStatement call while ADBC needs 1 ExecuteStatement and multiple FetchResults, which cause ADBC to be slower in small queries.
For ADBC:
<img width="614" height="136" alt="image" src="https://github.com/user-attachments/assets/64faa63c-9bc6-4dd1-8d71-66af09e95df4" />
For ODBC:
<img width="611" height="27" alt="image" src="https://github.com/user-attachments/assets/52817f46-412a-41fc-9f0b-17d7ae02d91d" />
This PR update the DefaultMaxBytes to 10MB, which is the same limit on Databricks backend for Arrow row set.
MaxRows to be 500K, assuming a minimum 20 Bytes column size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(csharp/src/Drivers/Databricks): Update DirectResult MaxRows MaxBytes setting [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #3489:
URL: https://github.com/apache/arrow-adbc/pull/3489#discussion_r2383549928
##########
csharp/test/Drivers/Databricks/E2E/DatabricksConnectionTest.cs:
##########
@@ -462,5 +462,27 @@ public void TracePropagationConfigurationTest(string tracePropagationEnabled, st
$"Connection created successfully with tracePropagationEnabled={tracePropagationEnabled}, " +
$"traceParentHeaderName={traceParentHeaderName}, traceStateEnabled={traceStateEnabled}");
}
+
+ /// <summary>
+ /// Tests that TrySetGetDirectResults uses DatabricksConnection's defaultGetDirectResults
+ /// </summary>
+ [Fact]
+ public void TrySetGetDirectResults_UsesDatabricksDefaultGetDirectResults()
+ {
+
Review Comment:
nit: remove blank line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(csharp/src/Drivers/Databricks): Update DirectResult MaxRows MaxBytes setting [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #3489:
URL: https://github.com/apache/arrow-adbc/pull/3489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org