You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alexguo-db (via GitHub)" <gi...@apache.org> on 2025/03/13 21:41:41 UTC
[PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
alexguo-db opened a new pull request, #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613
Enable trimming arrow batches to limit by default. This is the same setting in Databricks JDBC driver.
Tested locally with a program that executes a single query
```
dotnet run "SELECT * FROM main.schema.table LIMIT 11000"
Connected to Databricks successfully.
Read 4096 rows.
Read 4096 rows.
Read 2129 rows.
Read 679 rows.
Read 11000 rows.
```
Closes https://github.com/apache/arrow-adbc/issues/2612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#discussion_r1994591039
##########
csharp/src/Drivers/Apache/Spark/SparkStatement.cs:
##########
@@ -50,6 +50,7 @@ protected override void SetStatementProperties(TExecuteStatementReq statement)
ComplexTypesAsArrow = false,
IntervalTypesAsArrow = false,
};
+ statement.TrimArrowBatchesToLimit = true;
Review Comment:
And if it's not available in Spark (or in all relatively-recent versions of Spark), then what's the impact of setting this? Would it cause an error or be silently ignored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "alexguo-db (via GitHub)" <gi...@apache.org>.
alexguo-db commented on code in PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#discussion_r1995997917
##########
csharp/src/Drivers/Apache/Spark/SparkStatement.cs:
##########
@@ -50,6 +50,7 @@ protected override void SetStatementProperties(TExecuteStatementReq statement)
ComplexTypesAsArrow = false,
IntervalTypesAsArrow = false,
};
+ statement.TrimArrowBatchesToLimit = true;
Review Comment:
From what I researched it seems to be a Databricks only feature (I can't find any references in the Spark repo).
However I see that there are some other statement properties set here that I can't find in the open source Spark repo, e.g. UseArrowNativeTypes. Is there some place you are using to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "ccstevens (via GitHub)" <gi...@apache.org>.
ccstevens commented on PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#issuecomment-2725334630
We should not enable this. It should be strictly worse performance for cloud fetch. It causes Spark to do a read-modify-write of the last cloud fetch file. It is better for the client to simply read the last file and only process up to the result row count provided in the result manifest.
If this is being flipped on to make sure only the exact rows are returned for direct results, then we should go fix the bug causing direct results to return extra rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#discussion_r1996154059
##########
csharp/src/Drivers/Apache/Spark/SparkStatement.cs:
##########
@@ -50,6 +50,7 @@ protected override void SetStatementProperties(TExecuteStatementReq statement)
ComplexTypesAsArrow = false,
IntervalTypesAsArrow = false,
};
+ statement.TrimArrowBatchesToLimit = true;
Review Comment:
> Is there some place you are using to check?
It's possible that we haven't yet tested this against a (non-Databricks) Spark instance :(. We'll follow up. Finding any kind of definitive knowledge about this API (and its multiple implementations) is pretty challenging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "alexguo-db (via GitHub)" <gi...@apache.org>.
alexguo-db closed pull request #2613: feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default
URL: https://github.com/apache/arrow-adbc/pull/2613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "alexguo-db (via GitHub)" <gi...@apache.org>.
alexguo-db commented on PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#issuecomment-2725346126
> e should not enable this. It should be strictly worse performance for cloud fetch. It causes Spark to do a read-modify-write of the last cloud fetch file. It is better for the client to simply read the last file and only process up to the result row count provided in the result manifest.
>
> If this is being flipped on to make sure only the exact rows are returned for direct results, then we should go fix the bug causing direct results to return extra rows.
Ack, I will close this PR and corresponding issue. I will file a bug issue for direct result returning extra rows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "ccstevens (via GitHub)" <gi...@apache.org>.
ccstevens commented on code in PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#discussion_r1995997197
##########
csharp/src/Drivers/Apache/Spark/SparkStatement.cs:
##########
@@ -50,6 +50,7 @@ protected override void SetStatementProperties(TExecuteStatementReq statement)
ComplexTypesAsArrow = false,
IntervalTypesAsArrow = false,
};
+ statement.TrimArrowBatchesToLimit = true;
Review Comment:
Yes, this is Databricks only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Apache/Spark): Enable trimming arrow batches to limit by default [arrow-adbc]
Posted by "davidhcoe (via GitHub)" <gi...@apache.org>.
davidhcoe commented on code in PR #2613:
URL: https://github.com/apache/arrow-adbc/pull/2613#discussion_r1994401949
##########
csharp/src/Drivers/Apache/Spark/SparkStatement.cs:
##########
@@ -50,6 +50,7 @@ protected override void SetStatementProperties(TExecuteStatementReq statement)
ComplexTypesAsArrow = false,
IntervalTypesAsArrow = false,
};
+ statement.TrimArrowBatchesToLimit = true;
Review Comment:
is this only a Databricks feature or a general Spark feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org