You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/10/09 17:12:41 UTC

[PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

morningman opened a new pull request, #68847:
URL: https://github.com/apache/doris/pull/68847

   ### What problem does this PR solve?
   
   Issue Number: None
   
   Related PR: #68797, #66932
   
   Problem Summary:
   
   **In short.** The BE file cache can return the bytes of a different file for two kinds of external reads. An `http()` URL whose content changes keeps being served from blocks cached from the old content. Two object storage endpoints, or two HDFS clusters, that hold the same path with the same modification time share cache blocks. Both happen on master whenever a session turns on `enable_file_cache`, and on branch-4.x, where the cache is on by default. #68797 turns the cache on by default on master and is blocked on these two cases. This PR keeps HTTP responses out of the file cache and adds the storage to the cache key of external files.
   
   **Background.** `CachedRemoteFileReader` keys an external file as `path:mtime`: the path its reader reports plus the modification time from the file listing. The key is meant to change whenever the file does. `FileFactory` wraps S3, HDFS and HTTP readers in the cache when the query enables it. #66932 already exempted chunked HTTP responses (the `cdc_stream` table function), whose content is a stream.
   
   **The problem, and what it cost.**
   
   - **HTTP has no version.** `HttpTableValuedFunction` lists a URL with modification time 0, so every response of that URL gets the key `url:0`. When the content behind the URL changes, a later query reads blocks cached from the earlier content. If the length did not change, nothing fails and the query silently returns the old rows; if it did, old and new bytes can be mixed.
   - **The key does not name the storage.** An S3 reader reports `s3://bucket/key`, without the endpoint. An HDFS reader reports the path with the name node stripped (`convert_path`). Two S3-compatible endpoints (for example two MinIO deployments) holding the same bucket and key, or two HDFS clusters holding the same path, get the same key as soon as the modification times match. S3 reports them in seconds, so files written by the same job in the same second match. A query on one storage can then read blocks of the other storage's file.
   
   **How this PR fixes it.**
   
   - `FileFactory` no longer wraps HTTP readers in the file cache. This covers every HTTP source read through `FileFactory`, including `hf://` URLs, and replaces the chunked-response check from #66932, which this rule includes.
   - `FileReaderOptions` gets `fs_identity`, the storage an external file is read from. `FileFactory` sets it to the S3 endpoint (`AWS_ENDPOINT`) for S3 readers and to the name node (`fs_name`) for HDFS readers. `CachedRemoteFileReader` keys external files as `fs_identity:path:mtime`; readers that do not set it, such as broker reads, keep `path:mtime`.
   
   **Results.**
   
   | Case | Before | After |
   |---|---|---|
   | `http()` URL whose content was replaced, file cache on | second query can return rows of the old content | always reads the current content; HTTP data is not cached |
   | Same bucket, key and mtime on two S3-compatible endpoints | one cache entry, shared | one entry per endpoint |
   | Same path and mtime on two HDFS clusters | one cache entry, shared | one entry per name node |
   | Existing cached blocks of external S3 / HDFS files after upgrade | - | not found under the new key; the first reads go to the storage again and the old blocks age out |
   
   **Classes, and how they call each other.**
   
   ```
   FileScanner / format readers
     └─ FileFactory::create_file_reader(system_properties, file_description, reader_options)
          ├─ FILE_S3   : options.fs_identity = s3_conf.client_conf.endpoint ─┐
          ├─ FILE_HDFS : options.fs_identity = fs_name ──────────────────────┤
          │                                       create_cached_file_reader(reader, options)
          │                                         └─ CachedRemoteFileReader::_init_external_table_cache
          │                                              key = fs_identity:path:mtime (path:mtime if empty)
          └─ FILE_HTTP : HttpFileReader, not wrapped in the file cache
   ```
   
   ### Release note
   
   The file cache no longer keeps data read over HTTP (the `http()` table function, including `hf://` URLs), because an HTTP source has no modification time to tell versions apart. Cached data of external files on S3-compatible storage and HDFS is now kept per endpoint or name node; after upgrading, data cached by an earlier version is not reused for these files and is evicted over time.
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [x] Regression test
       - [x] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
       Regression test: `test_http_tvf` adds `replaced_before` / `replaced_after`. With `enable_file_cache = true`, the suite's local HTTP server replaces the file behind one URL with content of the same length between two queries, and the second query must return the new row. Run locally with the BE file cache on, it passes. With HTTP readers wrapped in the file cache again, the second query returns the old row (`1, aaaa` instead of `2, bbbb`). Only the two new blocks of `test_http_tvf.out` were taken from the generated file: a local regeneration also rewrites `sql08` (local time zone) and drops two blocks the suite no longer produces.
   
       BE unit tests:
       - `HttpFileReaderTest.PlainResponseIsNotFileCachedEither`: an HTTP reader is not wrapped in the file cache even when the query asks for the cache.
       - `AsyncCachedRemoteFileReaderTest.external_cache_key_tells_storages_apart`: the same path and modification time on two storages get different keys; a reader without an identity keeps `path:mtime`.
       - `AsyncCachedRemoteFileReaderTest.file_factory_keys_s3_files_by_endpoint`: readers that `FileFactory` builds for two S3 endpoints get different keys.
   
       `run-be-ut.sh --run --filter='HttpFileReaderTest.*:AsyncCachedRemoteFileReaderTest.*'` passed. Each new case fails when its part of the change is reverted: HTTP readers wrapped in the cache again (SEGV in `FileCacheFactory::get_by_path`, since that test has no cache), the S3 reader built without its endpoint, or the key built without the identity. The HDFS wiring has no unit test, because creating an HDFS reader in BE UT would start a JVM.
   
   - Behavior changed:
       - [ ] No.
       - [x] Yes. HTTP reads no longer use the file cache. External S3 / HDFS files are cached under a new key, so blocks cached before the upgrade are not reused.
   
   - Does this need documentation?
       - [x] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4237266695


##########
be/src/io/file_factory.cpp:
##########
@@ -248,10 +262,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
         }
         RETURN_IF_ERROR_RESULT(ExecEnv::GetInstance()->hdfs_mgr()->get_or_create_fs(
                 system_properties.hdfs_params, *fs_name, &handler));
+        // The reader's path drops the name node, so the cache key takes it from here.
+        auto options = with_storage_in_cache_key(reader_options, *fs_name);

Review Comment:
   I'd rather not key on the resolved HDFS configuration. To share a key, two clusters need the same nameservice ID, the same path and the same modification time, and HDFS keeps modification times in milliseconds. Files written independently do not meet that. The case that does is a copy made with preserved times (`distcp -pt`), which holds the same bytes, so the shared blocks are still correct. S3 was different because its timestamps are whole seconds, so files written by the same job in the same second matched. Keying on the NameNode addresses would also give one cluster several keys whenever catalogs spell its configuration differently (NameNode order, host name versus IP, an added NameNode), so ordinary reads would lose cache hits to cover a case that needs a deliberate copy.
   



##########
regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy:
##########
@@ -317,6 +324,27 @@ suite("test_http_tvf", "p0") {
         ) order by id;
     """
 
+    // The same URL serving new content of the same length: an HTTP response has no modification
+    // time for the file cache key, so the second query must not read blocks cached by the first.
+    sql """ set enable_file_cache = true """

Review Comment:
   Right: the p0 and External BE configurations leave `enable_file_cache` at its default, false, so in those pipelines the case checks the HTTP read but not the cache. #68797, which waits for this PR, turns the BE default on; from then on the same case covers the stale-cache path in every pipeline. Until then it was run locally with the BE cache on, where it fails when HTTP readers are wrapped in the cache again (the second query returns `1, aaaa`), and `HttpFileReaderTest.PlainResponseIsNotFileCachedEither` covers the reader decision in BE UT.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4232872565


##########
be/src/io/file_factory.cpp:
##########
@@ -232,10 +231,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
                 system_properties.properties, s3_uri, &s3_conf));
         auto client_holder = std::make_shared<io::ObjClientHolder>(s3_conf.client_conf);
         RETURN_IF_ERROR_RESULT(client_holder->init());
+        auto options = reader_options;
+        options.fs_identity = s3_conf.client_conf.endpoint;
         return io::S3FileReader::create(std::move(client_holder), s3_conf.bucket, s3_uri.get_key(),
                                         file_description.file_size, profile)
-                .and_then([&](auto&& reader) {
-                    return io::create_cached_file_reader(std::move(reader), reader_options);
+                .and_then([&options](auto&& reader) {
+                    return io::create_cached_file_reader(std::move(reader), options);

Review Comment:
   what about add an option called `cache_key_function`, which is a lambda to define the the cache key instead of tricking CacheRemoteFileReader



##########
be/src/io/file_factory.cpp:
##########
@@ -232,10 +231,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
                 system_properties.properties, s3_uri, &s3_conf));
         auto client_holder = std::make_shared<io::ObjClientHolder>(s3_conf.client_conf);
         RETURN_IF_ERROR_RESULT(client_holder->init());
+        auto options = reader_options;
+        options.fs_identity = s3_conf.client_conf.endpoint;
         return io::S3FileReader::create(std::move(client_holder), s3_conf.bucket, s3_uri.get_key(),
                                         file_description.file_size, profile)
-                .and_then([&](auto&& reader) {
-                    return io::create_cached_file_reader(std::move(reader), reader_options);
+                .and_then([&options](auto&& reader) {
+                    return io::create_cached_file_reader(std::move(reader), options);

Review Comment:
   what about adding an option called `cache_key_function`, which is a lambda to define the the cache key instead of tricking CacheRemoteFileReader



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6085666501

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6090145560

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68847_564e805bbebfcf8dbda003df9293416b35ce2ef6_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68847_564e805bbebfcf8dbda003df9293416b35ce2ef6_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 76.71% (35221/45912) |
   | Line Coverage     | 61.91% (398121/643026) |
   | Region Coverage   | 58.30% (335868/576071) |
   | Branch Coverage   | 59.21% (154533/261009) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6088436566

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 151939 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 564e805bbebfcf8dbda003df9293416b35ce2ef6, data reload: false
   
   query5	4297	587	465	465
   query6	427	205	190	190
   query7	4801	443	223	223
   query8	317	185	170	170
   query9	8750	3939	3913	3913
   query10	472	310	257	257
   query11	5888	3442	3189	3189
   query12	139	87	80	80
   query13	1245	427	323	323
   query14	6556	4683	4501	4501
   query14_1	4248	4247	4170	4170
   query15	210	203	184	184
   query16	932	437	403	403
   query17	869	672	538	538
   query18	2411	427	313	313
   query19	201	173	138	138
   query20	84	80	78	78
   query21	204	134	110	110
   query22	13061	13004	12998	12998
   query23	13252	12506	12249	12249
   query23_1	12385	12192	12209	12192
   query24	7104	831	462	462
   query24_1	479	478	476	476
   query25	544	435	366	366
   query26	1180	258	143	143
   query27	2737	438	268	268
   query28	4582	1928	1916	1916
   query29	1527	609	455	455
   query30	297	218	188	188
   query31	885	761	652	652
   query32	128	93	94	93
   query33	507	305	250	250
   query34	881	859	500	500
   query35	763	759	651	651
   query36	810	860	734	734
   query37	129	99	93	93
   query38	1791	1744	1678	1678
   query39	727	759	712	712
   query39_1	684	687	671	671
   query40	227	123	102	102
   query41	71	67	70	67
   query42	91	89	82	82
   query43	365	383	326	326
   query44	1289	691	700	691
   query45	174	187	162	162
   query46	860	958	560	560
   query47	2957	2996	2884	2884
   query48	294	308	216	216
   query49	575	436	287	287
   query50	661	271	215	215
   query51	10338	10263	10530	10263
   query52	81	77	72	72
   query53	182	208	160	160
   query54	238	203	177	177
   query55	67	67	63	63
   query56	237	210	212	210
   query57	1618	1548	1542	1542
   query58	280	263	244	244
   query59	2351	2385	2173	2173
   query60	282	232	218	218
   query61	148	151	144	144
   query62	394	340	281	281
   query63	196	156	174	156
   query64	2538	976	799	799
   query65	3417	3383	3388	3383
   query66	1735	418	296	296
   query67	20129	20168	19989	19989
   query68	3131	998	597	597
   query69	382	278	239	239
   query70	913	840	835	835
   query71	297	229	210	210
   query72	2714	2604	2317	2317
   query73	523	529	301	301
   query74	4577	4476	4298	4298
   query75	2338	2304	1944	1944
   query76	2302	1032	589	589
   query77	357	396	306	306
   query78	9016	8955	8493	8493
   query79	952	900	497	497
   query80	544	439	348	348
   query81	530	328	288	288
   query82	572	134	109	109
   query83	295	191	173	173
   query84	309	119	99	99
   query85	833	423	372	372
   query86	311	237	240	237
   query87	2020	1972	1825	1825
   query88	3596	2681	2650	2650
   query89	339	286	259	259
   query90	1910	171	169	169
   query91	165	148	120	120
   query92	94	84	76	76
   query93	992	991	555	555
   query94	456	316	283	283
   query95	581	339	385	339
   query96	670	518	237	237
   query97	2463	2438	2314	2314
   query98	156	146	141	141
   query99	774	764	658	658
   Total cold run time: 233072 ms
   Total hot run time: 151939 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6090106856

   # BE UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/564e805bbebfcf8dbda003df9293416b35ce2ef6_564e805bbebfcf8dbda003df9293416b35ce2ef6/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/564e805bbebfcf8dbda003df9293416b35ce2ef6_564e805bbebfcf8dbda003df9293416b35ce2ef6/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 65.24% (30947/47433) |
   | Line Coverage     | 50.04% (325461/650359) |
   | Region Coverage   | 45.48% (263066/578451) |
   | Branch Coverage   | 47.15% (123435/261808) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6094756638

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6088288708

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29136 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 564e805bbebfcf8dbda003df9293416b35ce2ef6, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17646	5713	5550	5550
   q2	2133	300	255	255
   q3	10225	1404	862	862
   q4	4675	469	359	359
   q5	7446	802	534	534
   q6	188	190	154	154
   q7	801	795	612	612
   q8	9246	1345	1071	1071
   q9	5790	4459	4451	4451
   q10	6814	1296	1040	1040
   q11	436	258	236	236
   q12	637	404	288	288
   q13	18402	3071	2391	2391
   q14	282	266	249	249
   q15	q16	750	731	692	692
   q17	1186	745	572	572
   q18	7143	6218	6188	6188
   q19	1117	957	589	589
   q20	376	344	227	227
   q21	5330	2504	2577	2504
   q22	389	354	312	312
   Total cold run time: 101012 ms
   Total hot run time: 29136 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6472	6492	6458	6458
   q2	740	605	574	574
   q3	5472	5672	5220	5220
   q4	2037	2155	1583	1583
   q5	5960	5761	5682	5682
   q6	248	192	153	153
   q7	2326	2054	1821	1821
   q8	3161	2711	2785	2711
   q9	8385	8149	7747	7747
   q10	3699	3601	3249	3249
   q11	605	400	391	391
   q12	670	696	501	501
   q13	2738	3070	2397	2397
   q14	279	293	262	262
   q15	q16	687	683	610	610
   q17	7775	7027	6927	6927
   q18	13170	12249	13165	12249
   q19	936	789	778	778
   q20	2208	2176	1954	1954
   q21	5754	4672	4796	4672
   q22	492	467	416	416
   Total cold run time: 73814 ms
   Total hot run time: 66355 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4233556674


##########
be/src/io/file_factory.cpp:
##########
@@ -232,10 +231,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
                 system_properties.properties, s3_uri, &s3_conf));
         auto client_holder = std::make_shared<io::ObjClientHolder>(s3_conf.client_conf);
         RETURN_IF_ERROR_RESULT(client_holder->init());
+        auto options = reader_options;
+        options.fs_identity = s3_conf.client_conf.endpoint;
         return io::S3FileReader::create(std::move(client_holder), s3_conf.bucket, s3_uri.get_key(),
                                         file_description.file_size, profile)
-                .and_then([&](auto&& reader) {
-                    return io::create_cached_file_reader(std::move(reader), reader_options);
+                .and_then([&options](auto&& reader) {
+                    return io::create_cached_file_reader(std::move(reader), options);

Review Comment:
   Thanks, done in 564e805bbeb. `FileReaderOptions` now has `cache_key_function`, which builds the key from the file's path and modification time. `FileFactory` sets it for S3 (endpoint) and HDFS (name node) readers, and `CachedRemoteFileReader` calls it when set and keeps `path:mtime` otherwise. The keys are the same as before. I also updated the PR description.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68847:
URL: https://github.com/apache/doris/pull/68847#discussion_r4236783493


##########
regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy:
##########
@@ -317,6 +324,27 @@ suite("test_http_tvf", "p0") {
         ) order by id;
     """
 
+    // The same URL serving new content of the same length: an HTTP response has no modification
+    // time for the file cache key, so the second query must not read blocks cached by the first.
+    sql """ set enable_file_cache = true """

Review Comment:
   [P2] Enable BE file caching for the mutable HTTP regression. This `SET` changes only the query option; `FileFactory::get_reader_options` also requires BE `config::enable_file_cache`, whose default is false and which the p0/external BE configurations do not set. In those runs both queries bypassed the cache even before this PR, so the new expected rows cannot detect a stale-cache regression. Run this case with a cache-enabled BE fixture (or an equivalent read-through unit test) and verify the second version's bytes.



##########
be/src/io/file_factory.cpp:
##########
@@ -248,10 +262,12 @@ Result<io::FileReaderSPtr> FileFactory::_create_file_reader_internal(
         }
         RETURN_IF_ERROR_RESULT(ExecEnv::GetInstance()->hdfs_mgr()->get_or_create_fs(
                 system_properties.hdfs_params, *fs_name, &handler));
+        // The reader's path drops the name node, so the cache key takes it from here.
+        auto options = with_storage_in_cache_key(reader_options, *fs_name);

Review Comment:
   [P1] Include the resolved HDFS configuration in this cache key. Two catalogs can use the same `hdfs://ns/path` while mapping `ns` to different NameNodes through `dfs.namenode.rpc-address.ns.*`; `HdfsMgr` creates distinct clients for their different `hdfs_conf`, but `*fs_name`, the reader's stripped path, and an equal mtime produce the same cache hash. A scan of the second cluster can therefore return the first cluster's cached blocks. Include a stable fingerprint of the effective HDFS connection in the key and cover this same-name/different-NameNode case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #68847:
URL: https://github.com/apache/doris/pull/68847


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6085666547

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6087098759

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.47 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit f1a604ca455ce7005617736bec6942aa330981b8, data reload: false
   
   query1	0.01	0.01	0.00
   query2	0.09	0.06	0.05
   query3	0.26	0.12	0.12
   query4	1.60	0.12	0.12
   query5	0.28	0.25	0.25
   query6	1.17	0.68	0.68
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.42	0.36	0.37
   query10	0.57	0.58	0.59
   query11	0.20	0.14	0.14
   query12	0.19	0.15	0.14
   query13	0.49	0.50	0.50
   query14	0.99	0.97	0.97
   query15	0.64	0.61	0.62
   query16	0.31	0.32	0.35
   query17	1.15	1.13	1.10
   query18	0.23	0.21	0.20
   query19	2.08	1.99	1.96
   query20	0.02	0.01	0.01
   query21	15.48	0.27	0.15
   query22	4.50	0.06	0.06
   query23	16.24	0.32	0.11
   query24	3.14	0.55	0.44
   query25	0.10	0.07	0.05
   query26	1.15	0.26	0.15
   query27	0.04	0.03	0.03
   query28	1.96	0.66	0.42
   query29	12.52	4.09	3.28
   query30	0.29	0.15	0.16
   query31	2.77	0.58	0.34
   query32	3.22	0.62	0.51
   query33	3.19	3.16	3.22
   query34	15.59	3.98	3.33
   query35	3.32	3.33	3.34
   query36	0.54	0.49	0.45
   query37	0.09	0.06	0.06
   query38	0.05	0.04	0.03
   query39	0.03	0.04	0.03
   query40	0.18	0.15	0.14
   query41	0.09	0.03	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.03	0.03
   Total cold run time: 95.36 s
   Total hot run time: 24.47 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6086874166

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29093 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit f1a604ca455ce7005617736bec6942aa330981b8, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17644	5637	5582	5582
   q2	2109	310	260	260
   q3	10249	1406	825	825
   q4	4671	476	353	353
   q5	7446	784	539	539
   q6	186	188	156	156
   q7	770	792	615	615
   q8	9236	1336	1071	1071
   q9	5695	4460	4454	4454
   q10	6829	1299	1029	1029
   q11	435	249	234	234
   q12	625	403	285	285
   q13	18050	3068	2398	2398
   q14	278	263	245	245
   q15	q16	722	724	669	669
   q17	1167	734	573	573
   q18	7044	6222	6167	6167
   q19	1104	956	591	591
   q20	372	329	229	229
   q21	5371	2538	2530	2530
   q22	375	355	288	288
   Total cold run time: 100378 ms
   Total hot run time: 29093 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6569	6457	6475	6457
   q2	686	604	601	601
   q3	5550	5708	5431	5431
   q4	2045	2142	1556	1556
   q5	5759	5711	5731	5711
   q6	245	194	152	152
   q7	2300	2087	1860	1860
   q8	3117	2743	2758	2743
   q9	8542	8051	7847	7847
   q10	3646	3585	3236	3236
   q11	623	408	392	392
   q12	702	694	499	499
   q13	2739	3072	2374	2374
   q14	283	287	266	266
   q15	q16	660	676	616	616
   q17	7861	6968	6958	6958
   q18	13077	12241	13124	12241
   q19	921	785	782	782
   q20	2169	2187	1961	1961
   q21	5758	4683	4759	4683
   q22	519	460	426	426
   Total cold run time: 73771 ms
   Total hot run time: 66792 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6087024436

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 151471 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit f1a604ca455ce7005617736bec6942aa330981b8, data reload: false
   
   query5	4305	602	449	449
   query6	420	222	191	191
   query7	4788	450	224	224
   query8	325	176	163	163
   query9	8772	3948	3936	3936
   query10	469	308	253	253
   query11	5999	3413	3194	3194
   query12	141	88	83	83
   query13	1248	442	300	300
   query14	6548	4751	4480	4480
   query14_1	4263	4236	4243	4236
   query15	206	202	182	182
   query16	937	454	431	431
   query17	821	641	533	533
   query18	2406	411	301	301
   query19	194	149	131	131
   query20	79	78	74	74
   query21	208	132	111	111
   query22	13026	13089	12791	12791
   query23	13083	12486	12104	12104
   query23_1	12237	12123	12163	12123
   query24	7107	829	458	458
   query24_1	449	466	452	452
   query25	516	428	329	329
   query26	1251	247	128	128
   query27	2787	434	264	264
   query28	4622	1898	1883	1883
   query29	1530	577	432	432
   query30	299	224	183	183
   query31	882	754	648	648
   query32	137	90	86	86
   query33	498	302	250	250
   query34	912	864	500	500
   query35	742	776	633	633
   query36	799	802	763	763
   query37	140	105	96	96
   query38	1798	1756	1695	1695
   query39	732	723	683	683
   query39_1	665	684	671	671
   query40	223	116	100	100
   query41	71	71	68	68
   query42	84	88	81	81
   query43	370	378	341	341
   query44	1299	692	705	692
   query45	184	190	169	169
   query46	848	965	572	572
   query47	2726	3115	2852	2852
   query48	301	313	224	224
   query49	574	402	301	301
   query50	707	285	233	233
   query51	10734	10612	10167	10167
   query52	84	83	70	70
   query53	190	208	161	161
   query54	246	196	215	196
   query55	74	72	68	68
   query56	243	234	230	230
   query57	1629	1589	1527	1527
   query58	287	263	250	250
   query59	2371	2388	2124	2124
   query60	295	273	239	239
   query61	171	171	174	171
   query62	398	347	294	294
   query63	194	167	171	167
   query64	2902	1062	869	869
   query65	3461	3359	3399	3359
   query66	1782	402	291	291
   query67	20290	20181	19920	19920
   query68	3049	995	601	601
   query69	394	306	253	253
   query70	933	844	835	835
   query71	286	224	207	207
   query72	2742	2377	2277	2277
   query73	504	520	293	293
   query74	4589	4495	4319	4319
   query75	2328	2275	1936	1936
   query76	2293	1023	628	628
   query77	368	390	299	299
   query78	9033	8967	8471	8471
   query79	952	903	516	516
   query80	567	412	348	348
   query81	535	333	275	275
   query82	571	134	113	113
   query83	273	200	182	182
   query84	308	126	104	104
   query85	805	426	371	371
   query86	311	246	233	233
   query87	2000	2010	1840	1840
   query88	3597	2689	2667	2667
   query89	348	296	258	258
   query90	1908	182	178	178
   query91	160	147	126	126
   query92	101	84	90	84
   query93	980	983	558	558
   query94	443	325	287	287
   query95	591	353	331	331
   query96	663	531	231	231
   query97	2461	2441	2316	2316
   query98	159	144	141	141
   query99	770	763	658	658
   Total cold run time: 233901 ms
   Total hot run time: 151471 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6087213413

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) Keep HTTP files out of the file cache and key external files by storage [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68847:
URL: https://github.com/apache/doris/pull/68847#issuecomment-6088506395

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.51 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 564e805bbebfcf8dbda003df9293416b35ce2ef6, data reload: false
   
   query1	0.01	0.01	0.00
   query2	0.10	0.04	0.04
   query3	0.27	0.12	0.12
   query4	1.61	0.12	0.12
   query5	0.24	0.24	0.24
   query6	1.16	0.67	0.69
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.45	0.35	0.36
   query10	0.60	0.62	0.56
   query11	0.19	0.14	0.13
   query12	0.18	0.14	0.14
   query13	0.48	0.49	0.48
   query14	1.00	0.97	0.98
   query15	0.66	0.62	0.62
   query16	0.36	0.36	0.34
   query17	1.20	1.13	1.13
   query18	0.22	0.21	0.22
   query19	2.03	1.94	1.97
   query20	0.02	0.01	0.02
   query21	15.49	0.27	0.15
   query22	4.48	0.06	0.05
   query23	16.28	0.32	0.13
   query24	3.20	0.65	0.44
   query25	0.13	0.06	0.04
   query26	1.18	0.26	0.16
   query27	0.04	0.03	0.03
   query28	1.77	0.57	0.39
   query29	12.50	4.16	3.25
   query30	0.28	0.13	0.14
   query31	2.77	0.56	0.34
   query32	3.23	0.62	0.53
   query33	3.18	3.18	3.20
   query34	15.62	4.07	3.41
   query35	3.38	3.36	3.37
   query36	0.55	0.45	0.41
   query37	0.10	0.07	0.07
   query38	0.05	0.04	0.04
   query39	0.03	0.04	0.04
   query40	0.17	0.16	0.15
   query41	0.09	0.03	0.03
   query42	0.03	0.03	0.04
   query43	0.04	0.03	0.04
   Total cold run time: 95.46 s
   Total hot run time: 24.51 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org