You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2025/10/20 03:20:12 UTC
[PR] Reduce number of object store requests when reading parquet files by … [datafusion]
zhuqi-lucas opened a new pull request, #18160:
URL: https://github.com/apache/datafusion/pull/18160
…default (set metadata_size_hint)
## Which issue does this PR close?
- Closes [#18118](https://github.com/apache/datafusion/issues/18118)
## Rationale for this change
Reduce number of object store requests when reading parquet files by default (set metadata_size_hint)
## What changes are included in this PR?
```rust
/// Default setting to 512 KB, which should be sufficient for most parquet files,
/// it can reduce one I/O operation per parquet file. If the metadata is larger than
/// the hint, two reads will still be performed.
pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
```
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce number of object store requests when reading parquet files by … [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2447016709
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
+ /// it can reduce one I/O operation per parquet file. If the metadata is larger than
+ /// the hint, two reads will still be performed.
+ pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
Review Comment:
cc @alamb for double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3429171331
@alamb Yes, agreed this should be a positive performance improvement on most datasets when using high latency storage, especially since fetching the parquet footer followed by the parquet metadata is a strictly sequential operation for each file.
The benchmark results here are a bit curious and look inconsistent (perhaps due to reasons out of everyone's control). However, I wouldn't be too surprised to see minor performance improvements from some local disk backed queries. The 8B fetch for the parquet footer is below pretty much any reasonable storage device's and file system's block size, so the local disk and filesystem are probably doing the same amount of work in either case, and this PR eliminates one extra call to disk and any internal runtime scheduling around managing that call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2465886794
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -120,24 +124,132 @@ async fn query_multi_csv_file() {
}
#[tokio::test]
-async fn create_single_parquet_file() {
+async fn create_single_parquet_file_default() {
+ // The default metadata size hint is 512KB
+ // which is enough to fetch the entire footer metadata and PageIndex
+ // in a single GET request.
+ let test = Test::new().with_single_file_parquet().await;
+ // expect 1 get request which reads the footer metadata and page index
assert_snapshot!(
- single_file_parquet_test().await.requests(),
+ test.requests(),
+ @r"
+ RequestCountingObjectStore()
+ Total Requests: 2
+ - HEAD path=parquet_table.parquet
+ - GET (range) range=0-2994 path=parquet_table.parquet
+ "
+ );
+}
+
+#[tokio::test]
+async fn create_single_parquet_file_prefetch() {
+ // Explicitly specify a prefetch hint that is adequate for the footer and page index
+ let test = Test::new()
+ .with_parquet_metadata_size_hint(Some(1000))
+ .with_single_file_parquet()
+ .await;
+ // expect 1 1000 byte request which reads the footer metadata and page index
+ assert_snapshot!(
+ test.requests(),
+ @r"
+ RequestCountingObjectStore()
+ Total Requests: 2
+ - HEAD path=parquet_table.parquet
+ - GET (range) range=1994-2994 path=parquet_table.parquet
+ "
+ );
+}
+
+#[tokio::test]
+async fn create_single_parquet_file_too_small_prefetch() {
+ // configure a prefetch size that is too small to fetch the footer
+ // metadata
+ //
+ // Using the ranges from the test below (with no_prefetch),
+ // pick a number less than 730:
+ // --------
+ // 2286-2294: (8 bytes) footer + length
+ // 2264-2986: (722 bytes) footer metadata
+ let test = Test::new()
+ .with_parquet_metadata_size_hint(Some(500))
+ .with_single_file_parquet()
+ .await;
+ // expect three get requests:
+ // 1. read the footer (500 bytes per hint, not enough for the footer metadata)
+ // 2. Read the footer metadata
+ // 3. reads the PageIndex
+ assert_snapshot!(
+ test.requests(),
@r"
RequestCountingObjectStore()
Total Requests: 4
- HEAD path=parquet_table.parquet
- - GET (range) range=2986-2994 path=parquet_table.parquet
+ - GET (range) range=2494-2994 path=parquet_table.parquet
- GET (range) range=2264-2986 path=parquet_table.parquet
- GET (range) range=2124-2264 path=parquet_table.parquet
"
);
}
+#[tokio::test]
+async fn create_single_parquet_file_small_prefetch() {
+ // configure a prefetch size that is large enough for the footer
+ // metadata but **not** the PageIndex
+ //
+ // Using the ranges from the test below (with no_prefetch),
+ // the 730 is determined as follows;
+ // --------
+ // 2286-2294: (8 bytes) footer + length
+ // 2264-2986: (722 bytes) footer metadata
+ let test = Test::new()
+ // 740 is enough to get both the footer + length (8 bytes)
+ // but not the entire PageIndex
+ .with_parquet_metadata_size_hint(Some(740))
+ .with_single_file_parquet()
+ .await;
+ // expect two get requests:
Review Comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce number of object store requests when reading parquet files by … [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2448420277
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
+ /// it can reduce one I/O operation per parquet file. If the metadata is larger than
+ /// the hint, two reads will still be performed.
+ pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
Review Comment:
will kick off benchmarks.
I think the potential downside of this approach is that it will make larger requests to objectstore / local disk by default and use slightly more memory for small files (it will always fetch / buffer 512K even if the actual footer is much smaller)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3455498901
I thnk this has been open enough -- let's merge it in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3433440503
I am working on some higher level tests for this feature (that show what object store requests are made for different operations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2465876294
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -120,24 +124,111 @@ async fn query_multi_csv_file() {
}
#[tokio::test]
-async fn create_single_parquet_file() {
+async fn create_single_parquet_file_default() {
Review Comment:
Great work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3427097124
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and issue_18118
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ issue_18118 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2660.79 ms │ 2663.08 ms │ no change │
│ QQuery 1 │ 1278.19 ms │ 1270.35 ms │ no change │
│ QQuery 2 │ 2378.56 ms │ 2432.55 ms │ no change │
│ QQuery 3 │ 1183.79 ms │ 1147.93 ms │ no change │
│ QQuery 4 │ 2257.26 ms │ 2222.80 ms │ no change │
│ QQuery 5 │ 27613.76 ms │ 27307.95 ms │ no change │
│ QQuery 6 │ 4120.87 ms │ 4159.21 ms │ no change │
│ QQuery 7 │ 3538.80 ms │ 3623.46 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 45032.00ms │
│ Total Time (issue_18118) │ 44827.32ms │
│ Average Time (HEAD) │ 5629.00ms │
│ Average Time (issue_18118) │ 5603.42ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ issue_18118 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.08 ms │ 2.44 ms │ 1.17x slower │
│ QQuery 1 │ 50.67 ms │ 50.74 ms │ no change │
│ QQuery 2 │ 137.10 ms │ 141.97 ms │ no change │
│ QQuery 3 │ 155.37 ms │ 163.15 ms │ 1.05x slower │
│ QQuery 4 │ 1030.04 ms │ 1063.64 ms │ no change │
│ QQuery 5 │ 1449.14 ms │ 1457.47 ms │ no change │
│ QQuery 6 │ 2.25 ms │ 2.13 ms │ +1.06x faster │
│ QQuery 7 │ 53.39 ms │ 56.36 ms │ 1.06x slower │
│ QQuery 8 │ 1420.49 ms │ 1450.88 ms │ no change │
│ QQuery 9 │ 1786.32 ms │ 1799.07 ms │ no change │
│ QQuery 10 │ 376.32 ms │ 381.83 ms │ no change │
│ QQuery 11 │ 428.31 ms │ 434.13 ms │ no change │
│ QQuery 12 │ 1327.53 ms │ 1373.14 ms │ no change │
│ QQuery 13 │ 2078.57 ms │ 2173.64 ms │ no change │
│ QQuery 14 │ 1244.16 ms │ 1278.07 ms │ no change │
│ QQuery 15 │ 1200.95 ms │ 1201.58 ms │ no change │
│ QQuery 16 │ 2597.14 ms │ 2642.62 ms │ no change │
│ QQuery 17 │ 2631.81 ms │ 2654.99 ms │ no change │
│ QQuery 18 │ 5589.64 ms │ 4913.85 ms │ +1.14x faster │
│ QQuery 19 │ 128.04 ms │ 128.14 ms │ no change │
│ QQuery 20 │ 1962.48 ms │ 2009.32 ms │ no change │
│ QQuery 21 │ 2298.54 ms │ 2302.72 ms │ no change │
│ QQuery 22 │ 9232.14 ms │ 3876.95 ms │ +2.38x faster │
│ QQuery 23 │ 23385.33 ms │ 12558.52 ms │ +1.86x faster │
│ QQuery 24 │ 223.35 ms │ 202.93 ms │ +1.10x faster │
│ QQuery 25 │ 483.34 ms │ 502.14 ms │ no change │
│ QQuery 26 │ 226.79 ms │ 213.12 ms │ +1.06x faster │
│ QQuery 27 │ 2895.97 ms │ 2790.24 ms │ no change │
│ QQuery 28 │ 24237.75 ms │ 24021.54 ms │ no change │
│ QQuery 29 │ 997.85 ms │ 987.09 ms │ no change │
│ QQuery 30 │ 1319.34 ms │ 1305.97 ms │ no change │
│ QQuery 31 │ 1331.19 ms │ 1327.00 ms │ no change │
│ QQuery 32 │ 5025.88 ms │ 4718.44 ms │ +1.07x faster │
│ QQuery 33 │ 5870.75 ms │ 5730.65 ms │ no change │
│ QQuery 34 │ 6056.12 ms │ 5917.27 ms │ no change │
│ QQuery 35 │ 1988.03 ms │ 1998.50 ms │ no change │
│ QQuery 36 │ 120.72 ms │ 120.39 ms │ no change │
│ QQuery 37 │ 51.24 ms │ 51.21 ms │ no change │
│ QQuery 38 │ 121.32 ms │ 120.68 ms │ no change │
│ QQuery 39 │ 194.35 ms │ 196.77 ms │ no change │
│ QQuery 40 │ 42.17 ms │ 41.52 ms │ no change │
│ QQuery 41 │ 40.17 ms │ 38.57 ms │ no change │
│ QQuery 42 │ 32.91 ms │ 31.74 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 111827.04ms │
│ Total Time (issue_18118) │ 94433.10ms │
│ Average Time (HEAD) │ 2600.63ms │
│ Average Time (issue_18118) │ 2196.12ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 33 │
│ Queries with Failure │ 0 │
└────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ issue_18118 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 172.88 ms │ 171.01 ms │ no change │
│ QQuery 2 │ 25.00 ms │ 26.96 ms │ 1.08x slower │
│ QQuery 3 │ 40.22 ms │ 39.88 ms │ no change │
│ QQuery 4 │ 27.54 ms │ 28.01 ms │ no change │
│ QQuery 5 │ 76.90 ms │ 77.15 ms │ no change │
│ QQuery 6 │ 19.43 ms │ 19.33 ms │ no change │
│ QQuery 7 │ 211.91 ms │ 214.86 ms │ no change │
│ QQuery 8 │ 31.48 ms │ 32.67 ms │ no change │
│ QQuery 9 │ 97.45 ms │ 103.98 ms │ 1.07x slower │
│ QQuery 10 │ 60.06 ms │ 59.55 ms │ no change │
│ QQuery 11 │ 16.72 ms │ 16.17 ms │ no change │
│ QQuery 12 │ 50.27 ms │ 51.00 ms │ no change │
│ QQuery 13 │ 45.02 ms │ 46.07 ms │ no change │
│ QQuery 14 │ 13.30 ms │ 13.27 ms │ no change │
│ QQuery 15 │ 24.08 ms │ 24.37 ms │ no change │
│ QQuery 16 │ 23.98 ms │ 24.59 ms │ no change │
│ QQuery 17 │ 144.63 ms │ 150.76 ms │ no change │
│ QQuery 18 │ 321.21 ms │ 321.15 ms │ no change │
│ QQuery 19 │ 36.44 ms │ 36.29 ms │ no change │
│ QQuery 20 │ 46.61 ms │ 47.90 ms │ no change │
│ QQuery 21 │ 325.99 ms │ 333.34 ms │ no change │
│ QQuery 22 │ 20.75 ms │ 20.26 ms │ no change │
└──────────────┴───────────┴─────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1831.87ms │
│ Total Time (issue_18118) │ 1858.56ms │
│ Average Time (HEAD) │ 83.27ms │
│ Average Time (issue_18118) │ 84.48ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└────────────────────────────┴───────────┘
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3427255070
> > 🤖: Benchmark completed
> > Details
>
> Interesting, it improved a lot for some cases even for local.
yeah that is unexpected -- let me rerun the benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2448441297
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
+ /// it can reduce one I/O operation per parquet file. If the metadata is larger than
+ /// the hint, two reads will still be performed.
+ pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
Review Comment:
Thanks, i agree if we have many small files.
> will kick off benchmarks.
>
> I think the potential downside of this approach is that it will make larger requests to objectstore / local disk by default and use slightly more memory for small files (it will always fetch / buffer 512K even if the actual footer is much smaller)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2465591625
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -121,8 +121,15 @@ async fn query_multi_csv_file() {
#[tokio::test]
async fn create_single_parquet_file() {
+ // Note that without a metadata size hint, the parquet reader
Review Comment:
I am a little confused about this -- I have some ideas and I am working on them now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #18160:
URL: https://github.com/apache/datafusion/pull/18160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2465630811
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -121,8 +121,15 @@ async fn query_multi_csv_file() {
#[tokio::test]
async fn create_single_parquet_file() {
+ // Note that without a metadata size hint, the parquet reader
Review Comment:
Thank you @alamb , i can't find a good way to use the parquet global option for register_parquet operation, so i added this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3433442277
(no need to hold up merging, but I do think we need better coverage to better illustrate the implications of this change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce number of object store requests when reading parquet files by … [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2446678689
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
+ /// it can reduce one I/O operation per parquet file. If the metadata is larger than
+ /// the hint, two reads will still be performed.
+ pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
Review Comment:
FWIW having some prefetch on as a default makes a ton of sense to me. I'd like to run the benchmarks to make sure it doesn't have a big impact, I'd guess no positive or negative impact since benchmarks run against local disc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3455499986
Thanks @zhuqi-lucas @BlakeOrth @adriangb and @xudong963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce number of object store requests when reading parquet files by … [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2448441297
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
+ /// it can reduce one I/O operation per parquet file. If the metadata is larger than
+ /// the hint, two reads will still be performed.
+ pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
Review Comment:
Thanks, i agree if we have many small files for local.
> will kick off benchmarks.
>
> I think the potential downside of this approach is that it will make larger requests to objectstore / local disk by default and use slightly more memory for small files (it will always fetch / buffer 512K even if the actual footer is much smaller)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3427256165
🤖 `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing issue_18118 (d7172b5eb6bd9628cbcefc0a6e45fc01f461c278) to 35b2e359f6703dedc6bf8ced3f0699d8a885d4df [diff](https://github.com/apache/datafusion/compare/35b2e359f6703dedc6bf8ced3f0699d8a885d4df..d7172b5eb6bd9628cbcefc0a6e45fc01f461c278) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3427484437
🤔 looks like a rough day on the benchmark farm. The second time the results for Q22 are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2451766545
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
Review Comment:
```suggestion
/// Default setting to 512 KiB, which should be sufficient for most parquet files,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce number of object store requests when reading parquet files by … [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2446702367
##########
datafusion/common/src/config.rs:
##########
@@ -620,7 +620,10 @@ config_namespace! {
/// bytes of the parquet file optimistically. If not specified, two reads are required:
/// One read to fetch the 8-byte parquet footer and
/// another to fetch the metadata length encoded in the footer
- pub metadata_size_hint: Option<usize>, default = None
+ /// Default setting to 512 KB, which should be sufficient for most parquet files,
+ /// it can reduce one I/O operation per parquet file. If the metadata is larger than
+ /// the hint, two reads will still be performed.
+ pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
Review Comment:
> FWIW having some prefetch on as a default makes a ton of sense to me. I'd like to run the benchmarks to make sure it doesn't have a big impact, I'd guess no positive or negative impact since benchmarks run against local disc.
Thank you @adriangb for review, i agree, it should not affect local disk performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3427364400
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and issue_18118
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ issue_18118 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2621.53 ms │ 2676.97 ms │ no change │
│ QQuery 1 │ 1245.30 ms │ 1306.64 ms │ no change │
│ QQuery 2 │ 2418.54 ms │ 2391.75 ms │ no change │
│ QQuery 3 │ 1194.39 ms │ 1154.42 ms │ no change │
│ QQuery 4 │ 2198.57 ms │ 2202.82 ms │ no change │
│ QQuery 5 │ 27624.59 ms │ 27556.71 ms │ no change │
│ QQuery 6 │ 4126.40 ms │ 4129.59 ms │ no change │
│ QQuery 7 │ 3383.00 ms │ 3311.29 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 44812.32ms │
│ Total Time (issue_18118) │ 44730.18ms │
│ Average Time (HEAD) │ 5601.54ms │
│ Average Time (issue_18118) │ 5591.27ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ issue_18118 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.20 ms │ 2.16 ms │ no change │
│ QQuery 1 │ 50.15 ms │ 50.46 ms │ no change │
│ QQuery 2 │ 133.84 ms │ 135.97 ms │ no change │
│ QQuery 3 │ 161.81 ms │ 164.63 ms │ no change │
│ QQuery 4 │ 1151.67 ms │ 1044.25 ms │ +1.10x faster │
│ QQuery 5 │ 1570.43 ms │ 1449.88 ms │ +1.08x faster │
│ QQuery 6 │ 2.13 ms │ 2.23 ms │ no change │
│ QQuery 7 │ 54.71 ms │ 55.95 ms │ no change │
│ QQuery 8 │ 1513.89 ms │ 1435.50 ms │ +1.05x faster │
│ QQuery 9 │ 1879.06 ms │ 1801.97 ms │ no change │
│ QQuery 10 │ 393.90 ms │ 388.31 ms │ no change │
│ QQuery 11 │ 437.86 ms │ 438.10 ms │ no change │
│ QQuery 12 │ 1498.20 ms │ 1336.94 ms │ +1.12x faster │
│ QQuery 13 │ 2207.79 ms │ 2119.97 ms │ no change │
│ QQuery 14 │ 1341.25 ms │ 1250.13 ms │ +1.07x faster │
│ QQuery 15 │ 1314.53 ms │ 1156.66 ms │ +1.14x faster │
│ QQuery 16 │ 2688.71 ms │ 2603.43 ms │ no change │
│ QQuery 17 │ 2658.89 ms │ 2594.40 ms │ no change │
│ QQuery 18 │ 5262.96 ms │ 4884.43 ms │ +1.08x faster │
│ QQuery 19 │ 127.33 ms │ 127.93 ms │ no change │
│ QQuery 20 │ 1966.44 ms │ 1954.50 ms │ no change │
│ QQuery 21 │ 2308.92 ms │ 2310.14 ms │ no change │
│ QQuery 22 │ 3908.92 ms │ 3901.61 ms │ no change │
│ QQuery 23 │ 14038.86 ms │ 12566.35 ms │ +1.12x faster │
│ QQuery 24 │ 229.12 ms │ 206.90 ms │ +1.11x faster │
│ QQuery 25 │ 501.17 ms │ 502.60 ms │ no change │
│ QQuery 26 │ 223.25 ms │ 211.17 ms │ +1.06x faster │
│ QQuery 27 │ 2894.88 ms │ 2816.34 ms │ no change │
│ QQuery 28 │ 24333.00 ms │ 24158.06 ms │ no change │
│ QQuery 29 │ 962.24 ms │ 969.41 ms │ no change │
│ QQuery 30 │ 1390.54 ms │ 1290.22 ms │ +1.08x faster │
│ QQuery 31 │ 1360.31 ms │ 1303.87 ms │ no change │
│ QQuery 32 │ 4358.47 ms │ 4570.94 ms │ no change │
│ QQuery 33 │ 5655.31 ms │ 5590.03 ms │ no change │
│ QQuery 34 │ 5824.96 ms │ 5867.01 ms │ no change │
│ QQuery 35 │ 2144.07 ms │ 1963.55 ms │ +1.09x faster │
│ QQuery 36 │ 123.88 ms │ 120.14 ms │ no change │
│ QQuery 37 │ 52.65 ms │ 52.88 ms │ no change │
│ QQuery 38 │ 122.74 ms │ 121.33 ms │ no change │
│ QQuery 39 │ 194.39 ms │ 198.62 ms │ no change │
│ QQuery 40 │ 42.62 ms │ 41.77 ms │ no change │
│ QQuery 41 │ 37.79 ms │ 37.69 ms │ no change │
│ QQuery 42 │ 32.30 ms │ 31.47 ms │ no change │
└──────────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 97158.13ms │
│ Total Time (issue_18118) │ 93829.86ms │
│ Average Time (HEAD) │ 2259.49ms │
│ Average Time (issue_18118) │ 2182.09ms │
│ Queries Faster │ 12 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 31 │
│ Queries with Failure │ 0 │
└────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ issue_18118 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 169.82 ms │ 168.96 ms │ no change │
│ QQuery 2 │ 25.41 ms │ 25.01 ms │ no change │
│ QQuery 3 │ 41.22 ms │ 39.13 ms │ +1.05x faster │
│ QQuery 4 │ 27.79 ms │ 28.34 ms │ no change │
│ QQuery 5 │ 76.28 ms │ 76.93 ms │ no change │
│ QQuery 6 │ 19.73 ms │ 19.36 ms │ no change │
│ QQuery 7 │ 214.22 ms │ 203.70 ms │ no change │
│ QQuery 8 │ 31.28 ms │ 32.27 ms │ no change │
│ QQuery 9 │ 96.93 ms │ 104.37 ms │ 1.08x slower │
│ QQuery 10 │ 58.74 ms │ 67.98 ms │ 1.16x slower │
│ QQuery 11 │ 16.09 ms │ 19.02 ms │ 1.18x slower │
│ QQuery 12 │ 49.64 ms │ 65.36 ms │ 1.32x slower │
│ QQuery 13 │ 46.26 ms │ 52.72 ms │ 1.14x slower │
│ QQuery 14 │ 13.43 ms │ 14.86 ms │ 1.11x slower │
│ QQuery 15 │ 23.83 ms │ 29.00 ms │ 1.22x slower │
│ QQuery 16 │ 23.94 ms │ 29.01 ms │ 1.21x slower │
│ QQuery 17 │ 142.84 ms │ 178.68 ms │ 1.25x slower │
│ QQuery 18 │ 319.56 ms │ 363.75 ms │ 1.14x slower │
│ QQuery 19 │ 36.28 ms │ 47.48 ms │ 1.31x slower │
│ QQuery 20 │ 47.33 ms │ 47.55 ms │ no change │
│ QQuery 21 │ 303.87 ms │ 316.45 ms │ no change │
│ QQuery 22 │ 20.39 ms │ 20.87 ms │ no change │
└──────────────┴───────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1804.88ms │
│ Total Time (issue_18118) │ 1950.79ms │
│ Average Time (HEAD) │ 82.04ms │
│ Average Time (issue_18118) │ 88.67ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 11 │
│ Queries with No Change │ 10 │
│ Queries with Failure │ 0 │
└────────────────────────────┴───────────┘
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3429047097
FYI @BlakeOrth -- I think this should also have a significant effect on the number of object store requests made by default (should reduce the number of requests by one for each file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3448585070
> > > * Add integration test for IO operations for listing tables queries [Add integration test for IO operations for listing tables queries #18229](https://github.com/apache/datafusion/pull/18229)
> >
> >
> > Great work @alamb , i will review it soon.
>
> Thanks to @zhuqi-lucas 's review I have merged the tests so I think we should be able to write a end to end test for this feature now
Thank you @alamb , added tests now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3448444294
> > * Add integration test for IO operations for listing tables queries [Add integration test for IO operations for listing tables queries #18229](https://github.com/apache/datafusion/pull/18229)
>
> Great work @alamb , i will review it soon.
Thanks to @zhuqi-lucas 's review I have merged the tests so I think we should be able to write a end to end test for this feature now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3444708409
Ok, I have now created a test harness that I think we can use to write a test that shows the effect of this change on access patterns:
- https://github.com/apache/datafusion/pull/18229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2465760442
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -120,24 +124,111 @@ async fn query_multi_csv_file() {
}
#[tokio::test]
-async fn create_single_parquet_file() {
+async fn create_single_parquet_file_default() {
Review Comment:
I reworked these tests a little to show the four scenarios:
* default prefetch_hint setting
* prefetch_hint setting that is not enough to read the metadata
* prefetch_hint setting that is enough to read the footer metadata but not page index
* prefetch_hint setting that is enough to read everything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#discussion_r2465772223
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -121,8 +121,15 @@ async fn query_multi_csv_file() {
#[tokio::test]
async fn create_single_parquet_file() {
+ // Note that without a metadata size hint, the parquet reader
Review Comment:
Yeah it was tricky -- I had to use `Option<Option<usize>>`` which is a bit 🤮 but I tried to make it clear with comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3427142950
> 🤖: Benchmark completed
>
> Details
Interesting, it improved a lot for some cases even for local.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce number of object store requests when reading parquet files by … [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3426799692
🤖 `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing issue_18118 (d7172b5eb6bd9628cbcefc0a6e45fc01f461c278) to 35b2e359f6703dedc6bf8ced3f0699d8a885d4df [diff](https://github.com/apache/datafusion/compare/35b2e359f6703dedc6bf8ced3f0699d8a885d4df..d7172b5eb6bd9628cbcefc0a6e45fc01f461c278) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Change default prefetch_hint to 512Kb to reduce number of object store requests when reading parquet files [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #18160:
URL: https://github.com/apache/datafusion/pull/18160#issuecomment-3445424767
> * Add integration test for IO operations for listing tables queries #18229
Great work @alamb , i will review it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org