You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/09 17:12:41 UTC
[PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
morningman opened a new pull request, #68847:
URL: https://github.com/apache/doris/pull/68847
### What problem does this PR solve?
Issue Number: None
Related PR: #68797, #66932
Problem Summary:
**In short.** The BE file cache can return the bytes of a different file for two kinds of external reads. An `http()` URL whose content changes keeps being served from blocks cached from the old content. Two object storage endpoints, or two HDFS clusters, that hold the same path with the same modification time share cache blocks. Both happen on master whenever a session turns on `enable_file_cache`, and on branch-4.x, where the cache is on by default. #68797 turns the cache on by default on master and is blocked on these two cases. This PR keeps HTTP responses out of the file cache and adds the storage to the cache key of external files.
**Background.** `CachedRemoteFileReader` keys an external file as `path:mtime`: the path its reader reports plus the modification time from the file listing. The key is meant to change whenever the file does. `FileFactory` wraps S3, HDFS and HTTP readers in the cache when the query enables it. #66932 already exempted chunked HTTP responses (the `cdc_stream` table function), whose content is a stream.
**The problem, and what it cost.**
- **HTTP has no version.** `HttpTableValuedFunction` lists a URL with modification time 0, so every response of that URL gets the key `url:0`. When the content behind the URL changes, a later query reads blocks cached from the earlier content. If the length did not change, nothing fails and the query silently returns the old rows; if it did, old and new bytes can be mixed.
- **The key does not name the storage.** An S3 reader reports `s3://bucket/key`, without the endpoint. An HDFS reader reports the path with the name node stripped (`convert_path`). Two S3-compatible endpoints (for example two MinIO deployments) holding the same bucket and key, or two HDFS clusters holding the same path, get the same key as soon as the modification times match. S3 reports them in seconds, so files written by the same job in the same second match. A query on one storage can then read blocks of the other storage's file.
**How this PR fixes it.**
- `FileFactory` no longer wraps HTTP readers in the file cache. This covers every HTTP source read through `FileFactory`, including `hf://` URLs, and replaces the chunked-response check from #66932, which this rule includes.
- `FileReaderOptions` gets `fs_identity`, the storage an external file is read from. `FileFactory` sets it to the S3 endpoint (`AWS_ENDPOINT`) for S3 readers and to the name node (`fs_name`) for HDFS readers. `CachedRemoteFileReader` keys external files as `fs_identity:path:mtime`; readers that do not set it, such as broker reads, keep `path:mtime`.
**Results.**
| Case | Before | After |
|---|---|---|
| `http()` URL whose content was replaced, file cache on | second query can return rows of the old content | always reads the current content; HTTP data is not cached |
| Same bucket, key and mtime on two S3-compatible endpoints | one cache entry, shared | one entry per endpoint |
| Same path and mtime on two HDFS clusters | one cache entry, shared | one entry per name node |
| Existing cached blocks of external S3 / HDFS files after upgrade | - | not found under the new key; the first reads go to the storage again and the old blocks age out |
**Classes, and how they call each other.**
```
FileScanner / format readers
└─ FileFactory::create_file_reader(system_properties, file_description, reader_options)
├─ FILE_S3 : options.fs_identity = s3_conf.client_conf.endpoint ─┐
├─ FILE_HDFS : options.fs_identity = fs_name ──────────────────────┤
│ create_cached_file_reader(reader, options)
│ └─ CachedRemoteFileReader::_init_external_table_cache
│ key = fs_identity:path:mtime (path:mtime if empty)
└─ FILE_HTTP : HttpFileReader, not wrapped in the file cache
```
### Release note
The file cache no longer keeps data read over HTTP (the `http()` table function, including `hf://` URLs), because an HTTP source has no modification time to tell versions apart. Cached data of external files on S3-compatible storage and HDFS is now kept per endpoint or name node; after upgrading, data cached by an earlier version is not reused for these files and is evicted over time.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Regression test: `test_http_tvf` adds `replaced_before` / `replaced_after`. With `enable_file_cache = true`, the suite's local HTTP server replaces the file behind one URL with content of the same length between two queries, and the second query must return the new row. Run locally with the BE file cache on, it passes. With HTTP readers wrapped in the file cache again, the second query returns the old row (`1, aaaa` instead of `2, bbbb`). Only the two new blocks of `test_http_tvf.out` were taken from the generated file: a local regeneration also rewrites `sql08` (local time zone) and drops two blocks the suite no longer produces.
BE unit tests:
- `HttpFileReaderTest.PlainResponseIsNotFileCachedEither`: an HTTP reader is not wrapped in the file cache even when the query asks for the cache.
- `AsyncCachedRemoteFileReaderTest.external_cache_key_tells_storages_apart`: the same path and modification time on two storages get different keys; a reader without an identity keeps `path:mtime`.
- `AsyncCachedRemoteFileReaderTest.file_factory_keys_s3_files_by_endpoint`: readers that `FileFactory` builds for two S3 endpoints get different keys.
`run-be-ut.sh --run --filter='HttpFileReaderTest.*:AsyncCachedRemoteFileReaderTest.*'` passed. Each new case fails when its part of the change is reverted: HTTP readers wrapped in the cache again (SEGV in `FileCacheFactory::get_by_path`, since that test has no cache), the S3 reader built without its endpoint, or the key built without the identity. The HDFS wiring has no unit test, because creating an HDFS reader in BE UT would start a JVM.
- Behavior changed:
- [ ] No.
- [x] Yes. HTTP reads no longer use the file cache. External S3 / HDFS files are cached under a new key, so blocks cached before the upgrade are not reused.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4237266695
##########
be/src/io/file_factory.cpp:
##########
@@ -248,10 +262,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
}
RETURN_IF_ERROR_RESULT(ExecEnv::GetInstance()->hdfs_mgr()->get_or_create_fs(
system_properties.hdfs_params, *fs_name, &handler));
+ // The reader's path drops the name node, so the cache key takes it from here.
+ auto options = with_storage_in_cache_key(reader_options, *fs_name);
Review Comment:
I'd rather not key on the resolved HDFS configuration. To share a key, two clusters need the same nameservice ID, the same path and the same modification time, and HDFS keeps modification times in milliseconds. Files written independently do not meet that. The case that does is a copy made with preserved times (`distcp -pt`), which holds the same bytes, so the shared blocks are still correct. S3 was different because its timestamps are whole seconds, so files written by the same job in the same second matched. Keying on the NameNode addresses would also give one cluster several keys whenever catalogs spell its configuration differently (NameNode order, host name versus IP, an added NameNode), so ordinary reads would lose cache hits to cover a case that needs a deliberate copy.
##########
regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy:
##########
@@ -317,6 +324,27 @@ suite("test_http_tvf", "p0") {
) order by id;
"""
+ // The same URL serving new content of the same length: an HTTP response has no modification
+ // time for the file cache key, so the second query must not read blocks cached by the first.
+ sql """ set enable_file_cache = true """
Review Comment:
Right: the p0 and External BE configurations leave `enable_file_cache` at its default, false, so in those pipelines the case checks the HTTP read but not the cache. #68797, which waits for this PR, turns the BE default on; from then on the same case covers the stale-cache path in every pipeline. Until then it was run locally with the BE cache on, where it fails when HTTP readers are wrapped in the cache again (the second query returns `1, aaaa`), and `HttpFileReaderTest.PlainResponseIsNotFileCachedEither` covers the reader decision in BE UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4232872565
##########
be/src/io/file_factory.cpp:
##########
@@ -232,10 +231,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
system_properties.properties, s3_uri, &s3_conf));
auto client_holder = std::make_shared<io::ObjClientHolder>(s3_conf.client_conf);
RETURN_IF_ERROR_RESULT(client_holder->init());
+ auto options = reader_options;
+ options.fs_identity = s3_conf.client_conf.endpoint;
return io::S3FileReader::create(std::move(client_holder), s3_conf.bucket, s3_uri.get_key(),
file_description.file_size, profile)
- .and_then([&](auto&& reader) {
- return io::create_cached_file_reader(std::move(reader), reader_options);
+ .and_then([&options](auto&& reader) {
+ return io::create_cached_file_reader(std::move(reader), options);
Review Comment:
what about add an option called `cache_key_function`, which is a lambda to define the the cache key instead of tricking CacheRemoteFileReader
##########
be/src/io/file_factory.cpp:
##########
@@ -232,10 +231,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
system_properties.properties, s3_uri, &s3_conf));
auto client_holder = std::make_shared<io::ObjClientHolder>(s3_conf.client_conf);
RETURN_IF_ERROR_RESULT(client_holder->init());
+ auto options = reader_options;
+ options.fs_identity = s3_conf.client_conf.endpoint;
return io::S3FileReader::create(std::move(client_holder), s3_conf.bucket, s3_uri.get_key(),
file_description.file_size, profile)
- .and_then([&](auto&& reader) {
- return io::create_cached_file_reader(std::move(reader), reader_options);
+ .and_then([&options](auto&& reader) {
+ return io::create_cached_file_reader(std::move(reader), options);
Review Comment:
what about adding an option called `cache_key_function`, which is a lambda to define the the cache key instead of tricking CacheRemoteFileReader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6085666501
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6090145560
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68847_564e805bbebfcf8dbda003df9293416b35ce2ef6_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68847_564e805bbebfcf8dbda003df9293416b35ce2ef6_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.71% (35221/45912) |
| Line Coverage | 61.91% (398121/643026) |
| Region Coverage | 58.30% (335868/576071) |
| Branch Coverage | 59.21% (154533/261009) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6088436566
<details>
<summary>TPC-DS: <b>Total hot run time: 151939 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 564e805bbebfcf8dbda003df9293416b35ce2ef6, data reload: false
query5 4297 587 465 465
query6 427 205 190 190
query7 4801 443 223 223
query8 317 185 170 170
query9 8750 3939 3913 3913
query10 472 310 257 257
query11 5888 3442 3189 3189
query12 139 87 80 80
query13 1245 427 323 323
query14 6556 4683 4501 4501
query14_1 4248 4247 4170 4170
query15 210 203 184 184
query16 932 437 403 403
query17 869 672 538 538
query18 2411 427 313 313
query19 201 173 138 138
query20 84 80 78 78
query21 204 134 110 110
query22 13061 13004 12998 12998
query23 13252 12506 12249 12249
query23_1 12385 12192 12209 12192
query24 7104 831 462 462
query24_1 479 478 476 476
query25 544 435 366 366
query26 1180 258 143 143
query27 2737 438 268 268
query28 4582 1928 1916 1916
query29 1527 609 455 455
query30 297 218 188 188
query31 885 761 652 652
query32 128 93 94 93
query33 507 305 250 250
query34 881 859 500 500
query35 763 759 651 651
query36 810 860 734 734
query37 129 99 93 93
query38 1791 1744 1678 1678
query39 727 759 712 712
query39_1 684 687 671 671
query40 227 123 102 102
query41 71 67 70 67
query42 91 89 82 82
query43 365 383 326 326
query44 1289 691 700 691
query45 174 187 162 162
query46 860 958 560 560
query47 2957 2996 2884 2884
query48 294 308 216 216
query49 575 436 287 287
query50 661 271 215 215
query51 10338 10263 10530 10263
query52 81 77 72 72
query53 182 208 160 160
query54 238 203 177 177
query55 67 67 63 63
query56 237 210 212 210
query57 1618 1548 1542 1542
query58 280 263 244 244
query59 2351 2385 2173 2173
query60 282 232 218 218
query61 148 151 144 144
query62 394 340 281 281
query63 196 156 174 156
query64 2538 976 799 799
query65 3417 3383 3388 3383
query66 1735 418 296 296
query67 20129 20168 19989 19989
query68 3131 998 597 597
query69 382 278 239 239
query70 913 840 835 835
query71 297 229 210 210
query72 2714 2604 2317 2317
query73 523 529 301 301
query74 4577 4476 4298 4298
query75 2338 2304 1944 1944
query76 2302 1032 589 589
query77 357 396 306 306
query78 9016 8955 8493 8493
query79 952 900 497 497
query80 544 439 348 348
query81 530 328 288 288
query82 572 134 109 109
query83 295 191 173 173
query84 309 119 99 99
query85 833 423 372 372
query86 311 237 240 237
query87 2020 1972 1825 1825
query88 3596 2681 2650 2650
query89 339 286 259 259
query90 1910 171 169 169
query91 165 148 120 120
query92 94 84 76 76
query93 992 991 555 555
query94 456 316 283 283
query95 581 339 385 339
query96 670 518 237 237
query97 2463 2438 2314 2314
query98 156 146 141 141
query99 774 764 658 658
Total cold run time: 233072 ms
Total hot run time: 151939 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6090106856
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/564e805bbebfcf8dbda003df9293416b35ce2ef6_564e805bbebfcf8dbda003df9293416b35ce2ef6/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/564e805bbebfcf8dbda003df9293416b35ce2ef6_564e805bbebfcf8dbda003df9293416b35ce2ef6/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 65.24% (30947/47433) |
| Line Coverage | 50.04% (325461/650359) |
| Region Coverage | 45.48% (263066/578451) |
| Branch Coverage | 47.15% (123435/261808) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6094756638
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6088288708
<details>
<summary>TPC-H: <b>Total hot run time: 29136 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 564e805bbebfcf8dbda003df9293416b35ce2ef6, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17646 5713 5550 5550
q2 2133 300 255 255
q3 10225 1404 862 862
q4 4675 469 359 359
q5 7446 802 534 534
q6 188 190 154 154
q7 801 795 612 612
q8 9246 1345 1071 1071
q9 5790 4459 4451 4451
q10 6814 1296 1040 1040
q11 436 258 236 236
q12 637 404 288 288
q13 18402 3071 2391 2391
q14 282 266 249 249
q15 q16 750 731 692 692
q17 1186 745 572 572
q18 7143 6218 6188 6188
q19 1117 957 589 589
q20 376 344 227 227
q21 5330 2504 2577 2504
q22 389 354 312 312
Total cold run time: 101012 ms
Total hot run time: 29136 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6472 6492 6458 6458
q2 740 605 574 574
q3 5472 5672 5220 5220
q4 2037 2155 1583 1583
q5 5960 5761 5682 5682
q6 248 192 153 153
q7 2326 2054 1821 1821
q8 3161 2711 2785 2711
q9 8385 8149 7747 7747
q10 3699 3601 3249 3249
q11 605 400 391 391
q12 670 696 501 501
q13 2738 3070 2397 2397
q14 279 293 262 262
q15 q16 687 683 610 610
q17 7775 7027 6927 6927
q18 13170 12249 13165 12249
q19 936 789 778 778
q20 2208 2176 1954 1954
q21 5754 4672 4796 4672
q22 492 467 416 416
Total cold run time: 73814 ms
Total hot run time: 66355 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4233556674
##########
be/src/io/file_factory.cpp:
##########
@@ -232,10 +231,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
system_properties.properties, s3_uri, &s3_conf));
auto client_holder = std::make_shared<io::ObjClientHolder>(s3_conf.client_conf);
RETURN_IF_ERROR_RESULT(client_holder->init());
+ auto options = reader_options;
+ options.fs_identity = s3_conf.client_conf.endpoint;
return io::S3FileReader::create(std::move(client_holder), s3_conf.bucket, s3_uri.get_key(),
file_description.file_size, profile)
- .and_then([&](auto&& reader) {
- return io::create_cached_file_reader(std::move(reader), reader_options);
+ .and_then([&options](auto&& reader) {
+ return io::create_cached_file_reader(std::move(reader), options);
Review Comment:
Thanks, done in 564e805bbeb. `FileReaderOptions` now has `cache_key_function`, which builds the key from the file's path and modification time. `FileFactory` sets it for S3 (endpoint) and HDFS (name node) readers, and `CachedRemoteFileReader` calls it when set and keeps `path:mtime` otherwise. The keys are the same as before. I also updated the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4236783493
##########
regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy:
##########
@@ -317,6 +324,27 @@ suite("test_http_tvf", "p0") {
) order by id;
"""
+ // The same URL serving new content of the same length: an HTTP response has no modification
+ // time for the file cache key, so the second query must not read blocks cached by the first.
+ sql """ set enable_file_cache = true """
Review Comment:
[P2] Enable BE file caching for the mutable HTTP regression. This `SET` changes only the query option; `FileFactory::get_reader_options` also requires BE `config::enable_file_cache`, whose default is false and which the p0/external BE configurations do not set. In those runs both queries bypassed the cache even before this PR, so the new expected rows cannot detect a stale-cache regression. Run this case with a cache-enabled BE fixture (or an equivalent read-through unit test) and verify the second version's bytes.
##########
be/src/io/file_factory.cpp:
##########
@@ -248,10 +262,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
}
RETURN_IF_ERROR_RESULT(ExecEnv::GetInstance()->hdfs_mgr()->get_or_create_fs(
system_properties.hdfs_params, *fs_name, &handler));
+ // The reader's path drops the name node, so the cache key takes it from here.
+ auto options = with_storage_in_cache_key(reader_options, *fs_name);
Review Comment:
[P1] Include the resolved HDFS configuration in this cache key. Two catalogs can use the same `hdfs://ns/path` while mapping `ns` to different NameNodes through `dfs.namenode.rpc-address.ns.*`; `HdfsMgr` creates distinct clients for their different `hdfs_conf`, but `*fs_name`, the reader's stripped path, and an equal mtime produce the same cache hash. A scan of the second cluster can therefore return the first cluster's cached blocks. Include a stable fingerprint of the effective HDFS connection in the key and cover this same-name/different-NameNode case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #68847:
URL: https://github.com/apache/doris/pull/68847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6085666547
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6087098759
<details>
<summary>ClickBench: <b>Total hot run time: 24.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f1a604ca455ce7005617736bec6942aa330981b8, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.06 0.05
query3 0.26 0.12 0.12
query4 1.60 0.12 0.12
query5 0.28 0.25 0.25
query6 1.17 0.68 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.42 0.36 0.37
query10 0.57 0.58 0.59
query11 0.20 0.14 0.14
query12 0.19 0.15 0.14
query13 0.49 0.50 0.50
query14 0.99 0.97 0.97
query15 0.64 0.61 0.62
query16 0.31 0.32 0.35
query17 1.15 1.13 1.10
query18 0.23 0.21 0.20
query19 2.08 1.99 1.96
query20 0.02 0.01 0.01
query21 15.48 0.27 0.15
query22 4.50 0.06 0.06
query23 16.24 0.32 0.11
query24 3.14 0.55 0.44
query25 0.10 0.07 0.05
query26 1.15 0.26 0.15
query27 0.04 0.03 0.03
query28 1.96 0.66 0.42
query29 12.52 4.09 3.28
query30 0.29 0.15 0.16
query31 2.77 0.58 0.34
query32 3.22 0.62 0.51
query33 3.19 3.16 3.22
query34 15.59 3.98 3.33
query35 3.32 3.33 3.34
query36 0.54 0.49 0.45
query37 0.09 0.06 0.06
query38 0.05 0.04 0.03
query39 0.03 0.04 0.03
query40 0.18 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 95.36 s
Total hot run time: 24.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6086874166
<details>
<summary>TPC-H: <b>Total hot run time: 29093 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f1a604ca455ce7005617736bec6942aa330981b8, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17644 5637 5582 5582
q2 2109 310 260 260
q3 10249 1406 825 825
q4 4671 476 353 353
q5 7446 784 539 539
q6 186 188 156 156
q7 770 792 615 615
q8 9236 1336 1071 1071
q9 5695 4460 4454 4454
q10 6829 1299 1029 1029
q11 435 249 234 234
q12 625 403 285 285
q13 18050 3068 2398 2398
q14 278 263 245 245
q15 q16 722 724 669 669
q17 1167 734 573 573
q18 7044 6222 6167 6167
q19 1104 956 591 591
q20 372 329 229 229
q21 5371 2538 2530 2530
q22 375 355 288 288
Total cold run time: 100378 ms
Total hot run time: 29093 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6569 6457 6475 6457
q2 686 604 601 601
q3 5550 5708 5431 5431
q4 2045 2142 1556 1556
q5 5759 5711 5731 5711
q6 245 194 152 152
q7 2300 2087 1860 1860
q8 3117 2743 2758 2743
q9 8542 8051 7847 7847
q10 3646 3585 3236 3236
q11 623 408 392 392
q12 702 694 499 499
q13 2739 3072 2374 2374
q14 283 287 266 266
q15 q16 660 676 616 616
q17 7861 6968 6958 6958
q18 13077 12241 13124 12241
q19 921 785 782 782
q20 2169 2187 1961 1961
q21 5758 4683 4759 4683
q22 519 460 426 426
Total cold run time: 73771 ms
Total hot run time: 66792 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6087024436
<details>
<summary>TPC-DS: <b>Total hot run time: 151471 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f1a604ca455ce7005617736bec6942aa330981b8, data reload: false
query5 4305 602 449 449
query6 420 222 191 191
query7 4788 450 224 224
query8 325 176 163 163
query9 8772 3948 3936 3936
query10 469 308 253 253
query11 5999 3413 3194 3194
query12 141 88 83 83
query13 1248 442 300 300
query14 6548 4751 4480 4480
query14_1 4263 4236 4243 4236
query15 206 202 182 182
query16 937 454 431 431
query17 821 641 533 533
query18 2406 411 301 301
query19 194 149 131 131
query20 79 78 74 74
query21 208 132 111 111
query22 13026 13089 12791 12791
query23 13083 12486 12104 12104
query23_1 12237 12123 12163 12123
query24 7107 829 458 458
query24_1 449 466 452 452
query25 516 428 329 329
query26 1251 247 128 128
query27 2787 434 264 264
query28 4622 1898 1883 1883
query29 1530 577 432 432
query30 299 224 183 183
query31 882 754 648 648
query32 137 90 86 86
query33 498 302 250 250
query34 912 864 500 500
query35 742 776 633 633
query36 799 802 763 763
query37 140 105 96 96
query38 1798 1756 1695 1695
query39 732 723 683 683
query39_1 665 684 671 671
query40 223 116 100 100
query41 71 71 68 68
query42 84 88 81 81
query43 370 378 341 341
query44 1299 692 705 692
query45 184 190 169 169
query46 848 965 572 572
query47 2726 3115 2852 2852
query48 301 313 224 224
query49 574 402 301 301
query50 707 285 233 233
query51 10734 10612 10167 10167
query52 84 83 70 70
query53 190 208 161 161
query54 246 196 215 196
query55 74 72 68 68
query56 243 234 230 230
query57 1629 1589 1527 1527
query58 287 263 250 250
query59 2371 2388 2124 2124
query60 295 273 239 239
query61 171 171 174 171
query62 398 347 294 294
query63 194 167 171 167
query64 2902 1062 869 869
query65 3461 3359 3399 3359
query66 1782 402 291 291
query67 20290 20181 19920 19920
query68 3049 995 601 601
query69 394 306 253 253
query70 933 844 835 835
query71 286 224 207 207
query72 2742 2377 2277 2277
query73 504 520 293 293
query74 4589 4495 4319 4319
query75 2328 2275 1936 1936
query76 2293 1023 628 628
query77 368 390 299 299
query78 9033 8967 8471 8471
query79 952 903 516 516
query80 567 412 348 348
query81 535 333 275 275
query82 571 134 113 113
query83 273 200 182 182
query84 308 126 104 104
query85 805 426 371 371
query86 311 246 233 233
query87 2000 2010 1840 1840
query88 3597 2689 2667 2667
query89 348 296 258 258
query90 1908 182 178 178
query91 160 147 126 126
query92 101 84 90 84
query93 980 983 558 558
query94 443 325 287 287
query95 591 353 331 331
query96 663 531 231 231
query97 2461 2441 2316 2316
query98 159 144 141 141
query99 770 763 658 658
Total cold run time: 233901 ms
Total hot run time: 151471 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6087213413
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6088506395
<details>
<summary>ClickBench: <b>Total hot run time: 24.51 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 564e805bbebfcf8dbda003df9293416b35ce2ef6, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.04 0.04
query3 0.27 0.12 0.12
query4 1.61 0.12 0.12
query5 0.24 0.24 0.24
query6 1.16 0.67 0.69
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.45 0.35 0.36
query10 0.60 0.62 0.56
query11 0.19 0.14 0.13
query12 0.18 0.14 0.14
query13 0.48 0.49 0.48
query14 1.00 0.97 0.98
query15 0.66 0.62 0.62
query16 0.36 0.36 0.34
query17 1.20 1.13 1.13
query18 0.22 0.21 0.22
query19 2.03 1.94 1.97
query20 0.02 0.01 0.02
query21 15.49 0.27 0.15
query22 4.48 0.06 0.05
query23 16.28 0.32 0.13
query24 3.20 0.65 0.44
query25 0.13 0.06 0.04
query26 1.18 0.26 0.16
query27 0.04 0.03 0.03
query28 1.77 0.57 0.39
query29 12.50 4.16 3.25
query30 0.28 0.13 0.14
query31 2.77 0.56 0.34
query32 3.23 0.62 0.53
query33 3.18 3.18 3.20
query34 15.62 4.07 3.41
query35 3.38 3.36 3.37
query36 0.55 0.45 0.41
query37 0.10 0.07 0.07
query38 0.05 0.04 0.04
query39 0.03 0.04 0.04
query40 0.17 0.16 0.15
query41 0.09 0.03 0.03
query42 0.03 0.03 0.04
query43 0.04 0.03 0.04
Total cold run time: 95.46 s
Total hot run time: 24.51 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org