You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jadewang-db (via GitHub)" <gi...@apache.org> on 2025/04/28 20:57:04 UTC
[PR] feat(csharp): fix powerbi hang when reading cloudfetch result in Databricks driver [arrow-adbc]
jadewang-db opened a new pull request, #2747:
URL: https://github.com/apache/arrow-adbc/pull/2747
### Summary
This PR fixes an issue where PowerBI would hang when reading CloudFetch results and significantly improves the logging capabilities in the CloudFetch downloader component.
### Problem
1. The CloudFetchReader was not properly disposing of the download manager after completing downloads, causing resource leaks that led to PowerBI hanging.
2. The CloudFetchDownloader was using Debug.WriteLine for logging, which is inadequate for production scenarios and doesn't provide sufficient diagnostic information.
### Solution
- Fixed resource management in CloudFetchReader by properly disposing the download manager after all files are processed
- Replaced Debug.WriteLine calls with more comprehensive Trace logging
- Added detailed performance metrics and diagnostics:
- Download start/completion timestamps
- File sizes and throughput calculations
- Decompression metrics
- Overall download statistics (total files, success/failure counts)
- Added URL sanitization for secure logging
- Added proper error tracking and reporting
### Testing
- Enhanced CloudFetchE2ETest to verify that the reader properly completes after all data is consumed
- Verified that PowerBI no longer hangs when reading CloudFetch results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp): fix powerbi hang when reading cloudfetch result in Databricks driver [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #2747:
URL: https://github.com/apache/arrow-adbc/pull/2747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp): fix powerbi hang when reading cloudfetch result in Databricks driver [arrow-adbc]
Posted by "eric-wang-1990 (via GitHub)" <gi...@apache.org>.
eric-wang-1990 commented on code in PR #2747:
URL: https://github.com/apache/arrow-adbc/pull/2747#discussion_r2064735879
##########
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloader.cs:
##########
@@ -274,14 +290,21 @@ private async Task DownloadFilesAsync(CancellationToken cancellationToken)
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// Expected when cancellation is requested
+ Trace.TraceInformation("Download process was cancelled");
}
catch (Exception ex)
{
- Debug.WriteLine($"Error in download loop: {ex.Message}");
+ Trace.TraceError($"Error in download loop: {ex.Message}");
SetError(ex);
}
finally
{
+ overallStopwatch.Stop();
+
+ Trace.TraceInformation(
Review Comment:
Does these log get exposed to PowerBI in anyway?
##########
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloader.cs:
##########
@@ -23,6 +23,7 @@
using System.Threading;
using System.Threading.Tasks;
using K4os.Compression.LZ4.Streams;
+using Microsoft.Extensions.Logging;
Review Comment:
Seems we are using Trace.TraceError, so this import is probably not used/needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp): fix powerbi hang when reading cloudfetch result in Databricks driver [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #2747:
URL: https://github.com/apache/arrow-adbc/pull/2747#discussion_r2065214730
##########
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloader.cs:
##########
@@ -298,6 +321,12 @@ private async Task DownloadFileAsync(IDownloadResult downloadResult, Cancellatio
// Use the size directly from the download result
long size = downloadResult.Size;
+ // Create a stopwatch to track download time
+ var stopwatch = Stopwatch.StartNew();
+
+ // Log download start
+ Trace.TraceInformation($"Starting download of file {downloadResult.Link.FileLink} from {SanitizeUrl(url)}, expected size: {size / 1024.0:F2} KB");
Review Comment:
All of these traces incorporate both the FileLink and a sanitized version of the url. But based on line 318, these are the same value. Is there a reason to log both an unsanitized version and a sanitized version of the same url in each trace message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(csharp): fix powerbi hang when reading cloudfetch result in Databricks driver [arrow-adbc]
Posted by "davidhcoe (via GitHub)" <gi...@apache.org>.
davidhcoe commented on code in PR #2747:
URL: https://github.com/apache/arrow-adbc/pull/2747#discussion_r2065169479
##########
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloader.cs:
##########
@@ -274,14 +290,21 @@ private async Task DownloadFilesAsync(CancellationToken cancellationToken)
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// Expected when cancellation is requested
+ Trace.TraceInformation("Download process was cancelled");
}
catch (Exception ex)
{
- Debug.WriteLine($"Error in download loop: {ex.Message}");
+ Trace.TraceError($"Error in download loop: {ex.Message}");
SetError(ex);
}
finally
{
+ overallStopwatch.Stop();
+
+ Trace.TraceInformation(
Review Comment:
Not currently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org