You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "BlakeOrth (via GitHub)" <gi...@apache.org> on 2025/10/13 22:36:08 UTC
[PR] Adds summary output to CLI instrumented object stores [datafusion]
BlakeOrth opened a new pull request, #18045:
URL: https://github.com/apache/datafusion/pull/18045
## Which issue does this PR close?
This does not fully close, but is an incremental building block component for:
- https://github.com/apache/datafusion/issues/17207
The full context of how this code is likely to progress can be seen in the POC for this effort:
- https://github.com/apache/datafusion/pull/17266
## Rationale for this change
For particularly large requests, in terms of number of objects in a table or large objects, the number of operations for a query may be quite large. In these cases, understanding the aggregate impact of various object store operations is likely the best way to understand the impact those operations had on a particular query. This PR allows users of an instrumented object store to understand and display basic summary statistics related to the `RequestDetails` collected during a query.
## What changes are included in this PR?
- Adds a `RequestSummary` type for the instrumented object store to display summary statistics about instrumented requests
- Adds a generic Stats type to track the statistics for the summary
- Adds tests for the new code
- Adds a basic summary output to the user-facing display when profiling is enabled
- Adds docs for new and newly exported public items
## Are these changes tested?
Yes. The new functionality has tests implemented, aside from testing the actual display output. The functional output can be seen below:
```sql
DataFusion CLI v50.1.0
> \object_store_profiling enabled
ObjectStore Profile mode set to Enabled
> CREATE EXTERNAL TABLE hits
STORED AS PARQUET
LOCATION 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
0 row(s) fetched.
Elapsed 0.268 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Enabled, inner: HttpStore
2025-10-13T22:15:50.518465131+00:00 operation=Get duration=0.030742s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-13T22:15:50.549263341+00:00 operation=Get duration=0.033060s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet
Summaries:
Get
count: 2
duration min: 0.030742s
duration max: 0.033060s
duration avg: 0.031901s
size min: 8 B
size max: 34322 B
size avg: 17165 B
size sum: 34330 B
>
```
## Are there any user-facing changes?
Yes? Just like the previous PR this does change the user-facing output, but there's no API breaking changes.
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds summary output to CLI instrumented object stores [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18045:
URL: https://github.com/apache/datafusion/pull/18045#issuecomment-3408422235
> @alamb Yeah, I'm not sure what the best way to display these is in a concise but readable manner. I think in the original issue for this I had also postulated that perhaps a table output, similar to the table information and query results might work.
>
> ```
> +----------+-----------+-----------+-----------+---------+
> | Metric | min | max | avg | sum |
> +----------+-----------+-----------+-----------+---------+
> | duration | 0.030742s | 0.033060s | 0.031901s | N/A |
> | size | 8 B | 34322 B | 17165 B | 34330 B |
> +----------+-----------+-----------+-----------+---------+
> ```
>
> Admittedly I'm probably not the right person to make that decision; UI/UX are _not_ my strong suit.
I made a PR with this
- https://github.com/apache/datafusion/pull/18085
It isn't ready for review yet but I'll get it ready tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds summary output to CLI instrumented object stores [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18045:
URL: https://github.com/apache/datafusion/pull/18045#discussion_r2430445956
##########
datafusion-cli/tests/snapshots/object_store_profiling@s3_url_fallback.snap:
##########
@@ -39,6 +39,17 @@ Object Store Profiling
Instrumented Object Store: instrument_mode: Enabled, inner: AmazonS3(data)
<TIMESTAMP> operation=Get duration=[DURATION] size=1006 path=cars.csv
+Summaries:
+Get
+count: 1
+[SUMMARY_DURATION]
Review Comment:
As a nice to have future feature, it would be great to have these values be more human readable -- e.g. in stead of `17446363 B` have it be `17.4 MB`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds summary output to CLI instrumented object stores [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on PR #18045:
URL: https://github.com/apache/datafusion/pull/18045#issuecomment-3403692516
@alamb Yeah, I'm not sure what the best way to display these is in a concise but readable manner. I think in the original issue for this I had also postulated that perhaps a table output, similar to the table information and query results might work.
```
+----------+-----------+-----------+-----------+---------+
| Metric | min | max | avg | sum |
+----------+-----------+-----------+-----------+---------+
| duration | 0.030742s | 0.033060s | 0.031901s | N/A |
| size | 8 B | 34322 B | 17165 B | 34330 B |
+----------+-----------+-----------+-----------+---------+
```
Admittedly I'm probably not the right person to make that decision; UI/UX are *not* my strong suit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds summary output to CLI instrumented object stores [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #18045:
URL: https://github.com/apache/datafusion/pull/18045
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds summary output to CLI instrumented object stores [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18045:
URL: https://github.com/apache/datafusion/pull/18045#issuecomment-3403604958
Another potential improvement is to reduce the number of lines used for display:
From:
```
Summaries:
Get
count: 2
duration min: 0.030742s
duration max: 0.033060s
duration avg: 0.031901s
size min: 8 B
size max: 34322 B
size avg: 17165 B
size sum: 34330 B
```
To something like
```
Summaries :
GET count: 2
duration(min/max/avg): 0.030742s / 0.033060s / 0.031901s
size(min/max/avg/sum): 8 B / 34322 B / 17165 B / 34330 B
```
but then again maybe that is even harder to read 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org