You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/10/15 21:46:29 UTC
[PR] Improve datafusion-cli object store profiling summary display [datafusion]
alamb opened a new pull request, #18085:
URL: https://github.com/apache/datafusion/pull/18085
Draft as I still need to polish it up (fix test normalization), but putting up what I have for feedback
## Which issue does this PR close?
- part of https://github.com/apache/datafusion/issues/17207
## Rationale for this change
As suggested by @BlakeOrth in https://github.com/apache/datafusion/pull/18045#issuecomment-3403692516 here is an attempt to improve the output of datafusion object store trace profiling:
## What changes are included in this PR?
Update the output format when `\object_store_profiling summary` is set
Current format
```sql
Summaries:
Get
count: 2
duration min: 0.024603s
duration max: 0.031946s
duration avg: 0.028274s
size min: 8 B
size max: 34322 B
size avg: 17165 B
size sum: 34330 B
```
New format (after this PR):
```sql
DataFusion CLI v50.2.0
> \object_store_profiling summary
ObjectStore Profile mode set to Summary
> select count(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
+----------+
| count(*) |
+----------+
| 1000000 |
+----------+
1 row(s) fetched.
Elapsed 0.548 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Summary, inner: HttpStore
Summaries:
+-----------+----------+-----------+-----------+-----------+-----------+
| Operation | Metric | min | max | avg | sum |
+-----------+----------+-----------+-----------+-----------+-----------+
| Get | duration | 0.040982s | 0.047195s | 0.044088s | 0.088176s |
| Get | size | 8 B | 34322 B | 17165 B | 34330 B |
+-----------+----------+-----------+-----------+-----------+-----------+
```
## Are these changes tested?
Yes
## Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2436711776
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -537,26 +662,14 @@ mod tests {
extra_display: None,
});
- let summaries = RequestSummary::summarize_by_operation(&requests);
- assert_eq!(summaries.len(), 1);
-
- let summary = summaries.get(&Operation::Get).unwrap();
- assert_eq!(summary.count, 1);
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().min,
- Duration::from_secs(5)
- );
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().max,
- Duration::from_secs(5)
- );
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().sum,
- Duration::from_secs(5)
- );
- assert_eq!(summary.size_stats.as_ref().unwrap().min, 100);
- assert_eq!(summary.size_stats.as_ref().unwrap().max, 100);
- assert_eq!(summary.size_stats.as_ref().unwrap().sum, 100);
+ assert_snapshot!(RequestSummaries::new(&requests), @r"
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ | Operation | Metric | min | max | avg | sum |
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ | Get | duration | 5.000000s | 5.000000s | 5.000000s | 5.000000s |
+ | Get | size | 100 B | 100 B | 100 B | 100 B |
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ ");
Review Comment:
Yeah, that's a good point. I think there was some discussion about building a machine-readable output, such as JSON, as well. That output would likely require its own set of test cases as it is though, so I think similar justification applies to both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2435929207
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -252,21 +265,109 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::fmt::Result {
+ // Format it like
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | Operation | Metric | min | max | avg | sum |
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | GET | duration | 0.030742s | 0.033060s | 0.031901s | N/A |
+ // | GET | size | 8 B | 34322 B | 17165 B | 34330 B |
+ // +------------+-----------+-----------+-----------+-----------+---------+
Review Comment:
Good call -- I have added the count column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2437066153
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -256,35 +264,165 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl fmt::Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let batch = self.to_batch();
+ write!(f, "{}", pretty_format_batches(&[batch]).unwrap())
Review Comment:
in [5d029d5](https://github.com/apache/datafusion/pull/18085/commits/5d029d5ddeedca3dbcbe9bdc6e08a957dcfd92a0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2434156249
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -252,21 +265,109 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::fmt::Result {
+ // Format it like
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | Operation | Metric | min | max | avg | sum |
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | GET | duration | 0.030742s | 0.033060s | 0.031901s | N/A |
+ // | GET | size | 8 B | 34322 B | 17165 B | 34330 B |
+ // +------------+-----------+-----------+-----------+-----------+---------+
Review Comment:
I'm not sure the best way to format it, but the table is missing the `count` statistic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2435947150
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -537,26 +662,14 @@ mod tests {
extra_display: None,
});
- let summaries = RequestSummary::summarize_by_operation(&requests);
- assert_eq!(summaries.len(), 1);
-
- let summary = summaries.get(&Operation::Get).unwrap();
- assert_eq!(summary.count, 1);
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().min,
- Duration::from_secs(5)
- );
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().max,
- Duration::from_secs(5)
- );
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().sum,
- Duration::from_secs(5)
- );
- assert_eq!(summary.size_stats.as_ref().unwrap().min, 100);
- assert_eq!(summary.size_stats.as_ref().unwrap().max, 100);
- assert_eq!(summary.size_stats.as_ref().unwrap().sum, 100);
+ assert_snapshot!(RequestSummaries::new(&requests), @r"
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ | Operation | Metric | min | max | avg | sum |
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ | Get | duration | 5.000000s | 5.000000s | 5.000000s | 5.000000s |
+ | Get | size | 100 B | 100 B | 100 B | 100 B |
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ ");
Review Comment:
I agree that now the test (implicitly) also relies on display / conversion behavior which might make it harder to track down
However, given the display output is also the end user API (aka what people will actually see) I think this is actually a good thing (it won't do any good if the summarizing logic is correct, but then it is not displayed correctly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#issuecomment-3417239639
Thanks @BlakeOrth and @comphead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2434171476
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -252,21 +265,109 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::fmt::Result {
+ // Format it like
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | Operation | Metric | min | max | avg | sum |
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | GET | duration | 0.030742s | 0.033060s | 0.031901s | N/A |
+ // | GET | size | 8 B | 34322 B | 17165 B | 34330 B |
+ // +------------+-----------+-----------+-----------+-----------+---------+
+
+ let operations: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| std::iter::repeat_n(Some(s.operation.to_string()), 2))
+ .collect();
+
+ let metrics: StringArray = std::iter::repeat([Some("duration"), Some("size")])
+ .flatten()
+ .take(self.summaries.len() * 2)
+ .collect();
+ let mins: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let dur_min = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ format!("{:.6}s", d.min.as_secs_f32())
+ });
+ let size_min = s
+ .size_stats
+ .as_ref()
+ .map_or("N/A".to_string(), |s| format!("{} B", s.min));
+ vec![Some(dur_min), Some(size_min)]
+ })
+ .collect();
+ let maxs: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let dur_max = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ format!("{:.6}s", d.max.as_secs_f32())
+ });
+ let size_max = s
+ .size_stats
+ .as_ref()
+ .map_or("N/A".to_string(), |s| format!("{} B", s.max));
+ vec![Some(dur_max), Some(size_max)]
+ })
+ .collect();
+ let avgs: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let count = s.count as f32;
+ let dur_avg = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ let avg = d.sum.as_secs_f32() / count;
+ format!("{:.6}s", avg)
+ });
+ let size_avg = s.size_stats.as_ref().map_or("N/A".to_string(), |s| {
+ let avg = s.sum as f32 / count;
+ format!("{} B", avg)
+ });
+ vec![Some(dur_avg), Some(size_avg)]
+ })
+ .collect();
+ let sums: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let dur_sum = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ format!("{:.6}s", d.sum.as_secs_f32())
Review Comment:
I specifically omitted a sum stat for `duration` in the initial implementation because I'm afraid it can be a bit misleading (at least at first glance). In my initial testing I found that for particularly large queries the sum of the durations was often larger than the total time of the query itself, which initially doesn't make much sense. The number for the sum is technically accurate, in the sense that it's the total "real" time across all CPU cores that participated in the IO operations, but communicating that technicality in a concise output like this is difficult.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#issuecomment-3408539384
This is a much improved way to display the summaries! I was agonizing over how to use Rust format padding etc. to build out the table, using a `RecordBatch` to get the formatted output for "free" is clever.
I have a PR queued up that instruments `LIST` that I was going to submit as soon as
- https://github.com/apache/datafusion/pull/18064
hits main (helps avoid resolving merge conflicts etc). Does it make sense to hold off finalizing these changes until we get `LIST` instrumented so we can ensure this behaves as expected with multiple operations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2435941727
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -252,21 +265,109 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::fmt::Result {
+ // Format it like
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | Operation | Metric | min | max | avg | sum |
+ // +------------+-----------+-----------+-----------+-----------+---------+
+ // | GET | duration | 0.030742s | 0.033060s | 0.031901s | N/A |
+ // | GET | size | 8 B | 34322 B | 17165 B | 34330 B |
+ // +------------+-----------+-----------+-----------+-----------+---------+
+
+ let operations: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| std::iter::repeat_n(Some(s.operation.to_string()), 2))
+ .collect();
+
+ let metrics: StringArray = std::iter::repeat([Some("duration"), Some("size")])
+ .flatten()
+ .take(self.summaries.len() * 2)
+ .collect();
+ let mins: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let dur_min = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ format!("{:.6}s", d.min.as_secs_f32())
+ });
+ let size_min = s
+ .size_stats
+ .as_ref()
+ .map_or("N/A".to_string(), |s| format!("{} B", s.min));
+ vec![Some(dur_min), Some(size_min)]
+ })
+ .collect();
+ let maxs: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let dur_max = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ format!("{:.6}s", d.max.as_secs_f32())
+ });
+ let size_max = s
+ .size_stats
+ .as_ref()
+ .map_or("N/A".to_string(), |s| format!("{} B", s.max));
+ vec![Some(dur_max), Some(size_max)]
+ })
+ .collect();
+ let avgs: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let count = s.count as f32;
+ let dur_avg = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ let avg = d.sum.as_secs_f32() / count;
+ format!("{:.6}s", avg)
+ });
+ let size_avg = s.size_stats.as_ref().map_or("N/A".to_string(), |s| {
+ let avg = s.sum as f32 / count;
+ format!("{} B", avg)
+ });
+ vec![Some(dur_avg), Some(size_avg)]
+ })
+ .collect();
+ let sums: StringArray = self
+ .summaries
+ .iter()
+ .flat_map(|s| {
+ let dur_sum = s.duration_stats.as_ref().map_or("N/A".to_string(), |d| {
+ format!("{:.6}s", d.sum.as_secs_f32())
Review Comment:
Removed (left blank)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #18085:
URL: https://github.com/apache/datafusion/pull/18085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#issuecomment-3416557711
Thank you for the review @comphead
> Thanks @alamb WDYT should it be documented?
@BlakeOrth did document the command here:
https://datafusion.apache.org/user-guide/cli/usage.html#commands
However, it seems to me like maybe an example in the docs would be good. I will add one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2434144273
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -537,26 +662,14 @@ mod tests {
extra_display: None,
});
- let summaries = RequestSummary::summarize_by_operation(&requests);
- assert_eq!(summaries.len(), 1);
-
- let summary = summaries.get(&Operation::Get).unwrap();
- assert_eq!(summary.count, 1);
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().min,
- Duration::from_secs(5)
- );
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().max,
- Duration::from_secs(5)
- );
- assert_eq!(
- summary.duration_stats.as_ref().unwrap().sum,
- Duration::from_secs(5)
- );
- assert_eq!(summary.size_stats.as_ref().unwrap().min, 100);
- assert_eq!(summary.size_stats.as_ref().unwrap().max, 100);
- assert_eq!(summary.size_stats.as_ref().unwrap().sum, 100);
+ assert_snapshot!(RequestSummaries::new(&requests), @r"
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ | Operation | Metric | min | max | avg | sum |
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ | Get | duration | 5.000000s | 5.000000s | 5.000000s | 5.000000s |
+ | Get | size | 100 B | 100 B | 100 B | 100 B |
+ +-----------+----------+-----------+-----------+-----------+-----------+
+ ");
Review Comment:
These test changes are the only thing that's really caused me to pause and think some. I really like how it's reduced the overall size of the test and improved readability. However, I think it comes with the tradeoff that _if_ there was a regression of some sort that caused the test to fail it's going to be a bit harder to hunt down exactly what's happening since it's now behind a display method implementation. I don't think I have a strong opinion on the "right" path forward here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2436731396
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -256,35 +264,165 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl fmt::Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let batch = self.to_batch();
+ write!(f, "{}", pretty_format_batches(&[batch]).unwrap())
Review Comment:
I suspect this `unwrap()` is generally safe since we're manually creating a pretty static `RecordBatch` for the input, however mapping the error since we can return a `Result`, or at least a comment related to panic safety, could be a nice addition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2436976418
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -256,35 +264,165 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl fmt::Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let batch = self.to_batch();
+ write!(f, "{}", pretty_format_batches(&[batch]).unwrap())
Review Comment:
Yes, I didn't like this either. I'll change the code to not panic on error. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve datafusion-cli object store profiling summary display [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18085:
URL: https://github.com/apache/datafusion/pull/18085#discussion_r2436976418
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -256,35 +264,165 @@ impl fmt::Display for RequestDetails {
}
}
-/// Summary statistics for an [`InstrumentedObjectStore`]'s [`RequestDetails`]
+/// Summary statistics for all requests recorded in an [`InstrumentedObjectStore`]
#[derive(Default)]
-pub struct RequestSummary {
- count: usize,
- duration_stats: Option<Stats<Duration>>,
- size_stats: Option<Stats<usize>>,
+pub struct RequestSummaries {
+ summaries: Vec<RequestSummary>,
}
-impl RequestSummary {
- /// Generates a set of [RequestSummaries](RequestSummary) from the input [`RequestDetails`]
- /// grouped by the input's [`Operation`]
- pub fn summarize_by_operation(
- requests: &[RequestDetails],
- ) -> HashMap<Operation, Self> {
- let mut summaries: HashMap<Operation, Self> = HashMap::new();
+/// Display the summary as a table
+impl fmt::Display for RequestSummaries {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let batch = self.to_batch();
+ write!(f, "{}", pretty_format_batches(&[batch]).unwrap())
Review Comment:
Yeah, I didn't like this either. I'll change the code to not panic on error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org