You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2025/10/30 03:37:36 UTC
[PR] feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads [arrow-adbc]
CurtHagenlocher opened a new pull request, #3652:
URL: https://github.com/apache/arrow-adbc/pull/3652
Improves memory utilization of cloud downloads by casting the downloaded and/or decompressed cloud data sets directly into Arrow data rather than having to deserialize them through a stream.
NOTE: I have not benchmarked this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads [arrow-adbc]
Posted by "eric-wang-1990 (via GitHub)" <gi...@apache.org>.
eric-wang-1990 commented on PR #3652:
URL: https://github.com/apache/arrow-adbc/pull/3652#issuecomment-3470931967
@CurtHagenlocher It seems this PR introduce a regression, I see this in the middle of a cloudfetch import:
Frown (Error)
Timestamp:
2025-10-30T15:19:38.4649362Z
Local Time:
2025-10-30T08:19:38.4649362-07:00
Session ID:
10423512-35fe-4714-af6b-d05dddbf2023
Release:
December 2025
Product Version:
2.150.0.0 (Main)+7c5552edc3b7e30ca4810aa7fea77f851cb4c03c (x64)
Error Message:
Corrupted IPC message. Received a continuation token at the end of the message.
Stack Trace:
Microsoft.Mashup.Host.Document.SerializedException
at Microsoft.Mashup.Evaluator.EvaluationHost.OnException(IEngineHost engineHost, IMessageChannel channel, ExceptionMessage message)
at Microsoft.Mashup.Evaluator.RemoteEvaluationContainerFactory.Container.OnException(IMessageChannel channel, ExceptionMessage message)
at Microsoft.Mashup.Evaluator.MessageHandlers.<>c__DisplayClass3_0`1.<AddHandler>b__0(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #3652:
URL: https://github.com/apache/arrow-adbc/pull/3652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on PR #3652:
URL: https://github.com/apache/arrow-adbc/pull/3652#issuecomment-3470988687
> @CurtHagenlocher It seems this PR introduce a regression,
That's odd, but I'll revert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on PR #3652:
URL: https://github.com/apache/arrow-adbc/pull/3652#issuecomment-3468134243
This turns out not to make much of a change other than reducing the number of garbage collections -- but that's worthwhile, and I think we can find other ways to further reduce that number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #3652:
URL: https://github.com/apache/arrow-adbc/pull/3652#discussion_r2476317029
##########
csharp/test/Drivers/Databricks/E2E/CloudFetch/CloudFetchDownloaderTest.cs:
##########
@@ -46,6 +47,7 @@ public CloudFetchDownloaderTest()
_resultQueue = new BlockingCollection<IDownloadResult>(new ConcurrentQueue<IDownloadResult>(), 10);
_mockMemoryManager = new Mock<ICloudFetchMemoryBufferManager>();
_mockStatement = new Mock<IHiveServer2Statement>();
+ _mockStatement.SetupGet(x => x.Trace).Returns(new ActivityTrace());
Review Comment:
A `null` value for `Trace` was preventing these tests from running correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org