You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "BlakeOrth (via GitHub)" <gi...@apache.org> on 2025/10/16 18:56:07 UTC
[PR] Adds instrumentation to LIST operations in CLI [datafusion]
BlakeOrth opened a new pull request, #18103:
URL: https://github.com/apache/datafusion/pull/18103
## Which issue does this PR close?
This does not fully close, but is an incremental building block component for:
- https://github.com/apache/datafusion/issues/17207
The full context of how this code is likely to progress can be seen in the POC for this effort:
- https://github.com/apache/datafusion/pull/17266
## Rationale for this change
Continued progress filling out the methods that are instrumented for the instrumented object store.
## What changes are included in this PR?
- Adds instrumentation around basic list operations into the instrumented object store
- Adds test cases for new code
## Are these changes tested?
Yes.
Example output:
```sql
DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET
LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet';
0 row(s) fetched.
Elapsed 2.679 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(altinity-clickhouse-data)
2025-10-16T18:53:09.512970085+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet
Summaries:
List
count: 1
Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(altinity-clickhouse-data)
2025-10-16T18:53:09.929709943+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet
2025-10-16T18:53:10.106757629+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet
2025-10-16T18:53:10.220555058+00:00 operation=Get duration=0.230604s size=8 range: bytes=222192975-222192982 path=nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet
2025-10-16T18:53:10.226399832+00:00 operation=Get duration=0.263826s size=8 range: bytes=233123927-233123934 path=nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet
2025-10-16T18:53:10.226194195+00:00 operation=Get duration=0.269754s size=8 range: bytes=252843253-252843260 path=nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet
. . .
2025-10-16T18:53:11.928787014+00:00 operation=Get duration=0.072248s size=18278 range: bytes=201384109-201402386 path=nyc_taxi_rides/data/tripdata_parquet/data-201509.parquet
2025-10-16T18:53:11.933475464+00:00 operation=Get duration=0.068880s size=17175 range: bytes=195411804-195428978 path=nyc_taxi_rides/data/tripdata_parquet/data-201601.parquet
2025-10-16T18:53:11.949629591+00:00 operation=Get duration=0.065645s size=19872 range: bytes=214807880-214827751 path=nyc_taxi_rides/data/tripdata_parquet/data-201603.parquet
Summaries:
List
count: 2
Get
count: 288
duration min: 0.060930s
duration max: 0.444601s
duration avg: 0.133339s
size min: 8 B
size max: 44247 B
size avg: 18870 B
size sum: 5434702 B
>
```
## Are there any user-facing changes?
No-ish
##
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18103:
URL: https://github.com/apache/datafusion/pull/18103#discussion_r2437209184
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -186,6 +209,10 @@ impl ObjectStore for InstrumentedObjectStore {
}
fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
+ if self.enabled() {
+ return self.instrumented_list(prefix);
+ }
+
Review Comment:
Yes, `list_with_delimiter` is "coming soon to a PR near you" just to help keep the size of the PRs easily reviewable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18103:
URL: https://github.com/apache/datafusion/pull/18103#discussion_r2437229437
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -138,6 +143,26 @@ impl InstrumentedObjectStore {
Ok(ret)
}
+
+ fn instrumented_list(
+ &self,
+ prefix: Option<&Path>,
+ ) -> BoxStream<'static, Result<ObjectMeta>> {
+ let timestamp = Utc::now();
+ let ret = self.inner.list(prefix);
+
+ self.requests.lock().push(RequestDetails {
+ op: Operation::List,
+ path: prefix.cloned().unwrap_or_else(|| Path::from("")),
+ timestamp,
+ duration: None, // list returns a stream, so the duration isn't meaningful
Review Comment:
Yes, not being able to easily evaluate a meaningful duration from this is a pretty big bummer honestly. I think time to first response is probably the ideal measurement to take here. I briefly looked into what it would take to make that happen within this instrumented store and I think it ends up being quite complex. I'm pretty sure we'd have to write a custom future to wrap the elements within the stream since the duration is only meaningful once elements in the stream start reporting `Poll::Ready`. Hopefully there's an easier way, because that sounds pretty painful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #18103:
URL: https://github.com/apache/datafusion/pull/18103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on PR #18103:
URL: https://github.com/apache/datafusion/pull/18103#issuecomment-3412526966
> I tested this locally, and I don't see any LIST appearing in the output 🤔
>
> I think you have to also instrument `list_wit_delimiter` 🤔
>
> ```sql
> andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ cargo run -p datafusion-cli
> Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.18s
> Running `target/debug/datafusion-cli`
> DataFusion CLI v50.2.0
> > \object_store_profiling trace
> ObjectStore Profile mode set to Trace
> > select count(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
> +----------+
> | count(*) |
> +----------+
> | 1000000 |
> +----------+
> 1 row(s) fetched.
> Elapsed 0.454 seconds.
>
> Object Store Profiling
> Instrumented Object Store: instrument_mode: Trace, inner: HttpStore
> 2025-10-16T19:01:11.225234+00:00 operation=Get duration=0.036593s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
> 2025-10-16T19:01:11.262391+00:00 operation=Get duration=0.036872s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet
>
> Summaries:
> Get
> count: 2
> duration min: 0.036593s
> duration max: 0.036872s
> duration avg: 0.036733s
> size min: 8 B
> size max: 34322 B
> size avg: 17165 B
> size sum: 34330 B
> ```
>
> EDIT: I do see it when I use `CREATE EXTERNAL TABLE`:
>
> ```sql
> Object Store Profiling
> Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(altinity-clickhouse-data)
> 2025-10-16T19:03:44.140517+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet
> 2025-10-16T19:03:44.268832+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet
> ```
So I think this is more or less expected. `https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet` is a single file behind an `https` endpoint, and as such there's no need (and in some sense no functional mechanism) for `ObjectStore`to list that `http` endpoint. I think we'd expect list to apply to buckets (like s3 in my example and your edit) and local directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on code in PR #18103:
URL: https://github.com/apache/datafusion/pull/18103#discussion_r2437229437
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -138,6 +143,26 @@ impl InstrumentedObjectStore {
Ok(ret)
}
+
+ fn instrumented_list(
+ &self,
+ prefix: Option<&Path>,
+ ) -> BoxStream<'static, Result<ObjectMeta>> {
+ let timestamp = Utc::now();
+ let ret = self.inner.list(prefix);
+
+ self.requests.lock().push(RequestDetails {
+ op: Operation::List,
+ path: prefix.cloned().unwrap_or_else(|| Path::from("")),
+ timestamp,
+ duration: None, // list returns a stream, so the duration isn't meaningful
Review Comment:
Yes, not being able to easily evaluate a meaningful duration from this is a pretty big bummer honestly. I think time to first response is probably the ideal measurement to take here. I briefly looked into what it would take to make that happen within this instrumented store and I think it ends up being quite complex. I'm pretty sure we'd have to write a custom future to wrap the elements within the stream since the duration is only meaningful once elements in the stream start reporting `Poll:Ready`. Hopefully there's an easier way, because that sounds pretty painful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18103:
URL: https://github.com/apache/datafusion/pull/18103#discussion_r2440731675
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -138,6 +143,26 @@ impl InstrumentedObjectStore {
Ok(ret)
}
+
+ fn instrumented_list(
+ &self,
+ prefix: Option<&Path>,
+ ) -> BoxStream<'static, Result<ObjectMeta>> {
+ let timestamp = Utc::now();
+ let ret = self.inner.list(prefix);
+
+ self.requests.lock().push(RequestDetails {
+ op: Operation::List,
+ path: prefix.cloned().unwrap_or_else(|| Path::from("")),
+ timestamp,
+ duration: None, // list returns a stream, so the duration isn't meaningful
Review Comment:
- Filed https://github.com/apache/datafusion/issues/18138 to track
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Adds instrumentation to LIST operations in CLI [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18103:
URL: https://github.com/apache/datafusion/pull/18103#discussion_r2437136357
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -138,6 +143,26 @@ impl InstrumentedObjectStore {
Ok(ret)
}
+
+ fn instrumented_list(
+ &self,
+ prefix: Option<&Path>,
+ ) -> BoxStream<'static, Result<ObjectMeta>> {
+ let timestamp = Utc::now();
+ let ret = self.inner.list(prefix);
+
+ self.requests.lock().push(RequestDetails {
+ op: Operation::List,
+ path: prefix.cloned().unwrap_or_else(|| Path::from("")),
+ timestamp,
+ duration: None, // list returns a stream, so the duration isn't meaningful
Review Comment:
Time to first response is meaningful 🤔 Something we can improve in the future if needed
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -172,9 +197,7 @@ impl ObjectStore for InstrumentedObjectStore {
}
async fn get_opts(&self, location: &Path, options: GetOptions) -> Result<GetResult> {
- if self.instrument_mode.load(Ordering::Relaxed)
- != InstrumentedObjectStoreMode::Disabled as u8
- {
+ if self.enabled() {
Review Comment:
❤️
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -186,6 +209,10 @@ impl ObjectStore for InstrumentedObjectStore {
}
fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
+ if self.enabled() {
+ return self.instrumented_list(prefix);
+ }
+
Review Comment:
I think you may need to instrument the other versions of list below (`list_with_delimiter`, etc) as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org