You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/13 05:21:45 UTC

[PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Gabriel39 opened a new pull request, #68998:
URL: https://github.com/apache/doris/pull/68998

   ### What problem does this PR solve?
   
   External connector queries can fail during catalog initialization, file listing, or partition pruning after dependency isolation. This fixes the following paths:
   
   - Keep DLF and Paimon OSS adapters with their plugin APIs, isolate the Hive expression implementation required by DLF, and share the host Jindo SDK/native runtime. Use the DLF client's actual constructor signature, and preserve standard Hive storage-format class names in persisted metadata.
   - Normalize numeric ranges for Hadoop file listing and recognize legacy OSS authorities while preserving raw object keys and the configured endpoint.
   - Resolve ambiguous escaped Hudi partition names through their storage locations, keep TIMESTAMPTZ predicates for typed filtering, and pass the session timezone when BE parses offset-free partition values.
   - Continue MaxCompute schema enumeration when a listed schema disappears during its lazy reload, while preserving other errors.
   
   The QL isolation serves DLF's local expression-proxy initialization. Current connector metadata operations use partition-name APIs; private Kryo expressions are not treated as an external wire format.
   
   ### Release note
   
   Fix external catalog initialization and file/partition discovery failures affecting DLF, Paimon OSS, Hudi, Hadoop glob paths, legacy OSS locations, and concurrent MaxCompute schema deletion.
   
   ### Check List (For Author)
   
   - Test
       - [x] Unit Test
       - [x] Manual test (isolated packaged plugins with a shared Jindo runtime)
   
   - Behavior changed:
       - [x] Yes. Previously failing connector paths now preserve the expected catalog, file, and partition semantics. Numeric glob expansion is bounded to prevent excessive allocation.
   
   - Does this need documentation?
       - [x] No.
   
   ### Validation
   
   - Connector module suites and packages: HMS 155 tests, Hive 491, Hudi 255, MaxCompute 155 (1 skipped), Iceberg 1,527 (5 skipped), and Paimon 726 (1 skipped); zero failures or errors in the final runs.
   - Filesystem compatibility suites: 153 tests passed, including real Hadoop local-filesystem glob expansion and captured OSS requests.
   - BE ASAN: all 29 `FileScannerV2Test` tests passed. Without the production fix, the new timestamp-partition test fails and the other 28 pass.
   - Regression tests cover isolated expression-proxy loading, offline real DLF construction, OSS SPI discovery, canonical HMS storage-format metadata, schema deletion races, escaped Hudi paths, typed timestamp residuals, and explicit-offset precedence.
   - Actual plugin ZIPs with a host Hive 3 QL jar: both offline DLF constructors, OSS SPI discovery, adapter/interface ownership, canonical format names, shared Jindo classes, and native initialization through both connector loaders passed.
   - FE Checkstyle, clang-format 16, changed-line clang-tidy, build hygiene, shell syntax, and diff whitespace checks passed.
   
   The remote external regression suites require external services and were not rerun locally. Standalone Hive Kryo serialization against the host QL jar still encounters a host `HiveConf` dependency gap; caller and vendor-bytecode checks confirm the affected name-based metadata paths do not invoke it. This PR does not claim to enable that separate path.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6182627341

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6140433361

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6146708671

   # FE UT Coverage Report
   Increment line coverage `` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_77483a190e1adec4a2b0a53339d2cceeddbe262b/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_77483a190e1adec4a2b0a53339d2cceeddbe262b/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4257554103


##########
fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java:
##########
@@ -277,7 +279,7 @@ public List<FileEntry> listFiles(Location location) throws IOException {
     }
 
     private static boolean containsGlob(String path) {
-        return path.contains("*") || path.contains("?") || path.contains("[");
+        return path.contains("*") || path.contains("?") || path.contains("[") || path.contains("{");

Review Comment:
   Fixed in b3039f6f488. The cleanup consumer now enumerates the literal target using FileSystem.list, filters out directories, closes the iterator, and then removes the collected files. It retains the staging-directory exclusion. I audited the other callers before choosing this approach: backup and Parquet metadata still rely on listFiles glob behavior, so removing that behavior would introduce a separate regression.
   
   Added an actual HiveConnectorTransaction regression over DFSFileSystem/Hadoop local storage with relative staging and table directory names containing {}, *, ? and []. It verifies the old file is removed, the new file is committed, and a neighboring glob-matching directory remains untouched. The brace-directory test failed before the fix. All existing filesystem discovery tests remain passing.



##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -794,7 +799,8 @@ private List<ConnectorPartitionInfo> collectPartitions(HudiTableHandle handle, b
                 // HoodieTableMetaClient built just to read a timeline zone - plus a fallback that
                 // would have published epoch 0 as a partition's last-modified time for any
                 // non-numeric pin, the day it became reachable.
-                return buildPartitionInfos(hmsNames, partKeyNames, latestInstantMillis(handle));
+                return buildPartitionInfos(hmsPartitionPaths(handle, hmsNames),

Review Comment:
   Fixed in b3039f6f488. Physical-path parsing now takes an explicit encoding flag from HoodieTableConfig.getUrlEncodePartitioning(). Missing/false preserves literal percent sequences; true unescapes once. The flag is propagated through listing, pruning, pinned-partition filtering, snapshot scans and COW/MOR incremental scans, so the displayed values and scan values use the same contract. HMS name decoding for partition-object identity is kept separate.
   
   Added tests using real persisted hoodie.properties for missing, false and true settings. They distinguish raw ':' and '%3A' in both listPartitions/listPartitionNames and applyFilter; the default-setting test reproduced the colon collision before the fix. Additional tests cover metadata/pinned listings, positional and prefixed paths, percent/slash/plus values, and MOR native/JNI ranges. The final scoped run passed 906 tests with no failures, errors or skips; FE Checkstyle passed for all 23 reactor modules.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4255125714


##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+    private static final int MAX_EXPANSIONS = 10000;
+    private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+    private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+    private static final Pattern BRACE_GROUP = Pattern.compile("\\{([^{}]*)\\}");

Review Comment:
   [P2] Preserve escaped braces inside numeric alternatives
   
   For the valid pattern `data_{1..2,\{special\}}.csv`, this regex cannot match the outer group because its content includes the escaped opening brace. It matches only `\{special\}`, which the escape check then skips, leaving `1..2` unexpanded. `S3CompatibleFileSystem.compileGlobPattern` and `globListWithLimit` consequently match literal `data_1..2.csv` instead of `data_1.csv` and `data_2.csv`; the previous normalizer expanded this pattern correctly. Parse brace boundaries with escape awareness so literal braces in one alternative do not suppress ranges in another, and add this mixed escaped-brace case to the coverage.



##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +927,42 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
         return atPin;
     }
 
+    private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+        List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+                .collect(Collectors.toList());
+        if (encodedNames.isEmpty()) {
+            return names;
+        }
+        // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+        // The storage location identifies the physical path regardless of the sync extractor's decoding
+        // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+        List<HmsPartitionInfo> partitions = hmsClient.getPartitions(

Review Comment:
   [P2] Tolerate partitions deleted after the cached name listing
   
   With `use_hive_sync_partition=true` and `meta.cache.hive.partition.enable=false` (the name cache remains enabled), cache a listing containing an encoded partition, then drop that partition in HMS. The next filtered query for a surviving partition still passes the stale encoded name here, and strict `getPartitions` throws for the missing result before `prunePartitionPaths` can discard it. Object-cache expiry or eviction causes the same failure, so retries can keep failing until the names cache is refreshed. Use an omission-tolerant lookup such as `getExistingPartitions` and associate surviving results by partition identity, omitting deleted names; retain errors for malformed results. Add a cached-name/deleted-partition case to the tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141741467

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29362 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 2cfee92005dbc607c8ac31897868be045670ef6f, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17630	5684	5593	5593
   q2	2152	335	249	249
   q3	10213	1429	853	853
   q4	4678	481	352	352
   q5	7451	788	543	543
   q6	189	191	166	166
   q7	774	799	630	630
   q8	9267	1345	1085	1085
   q9	5726	4491	4465	4465
   q10	6833	1295	1039	1039
   q11	429	254	241	241
   q12	634	414	290	290
   q13	18077	3106	2409	2409
   q14	281	273	254	254
   q15	q16	756	733	688	688
   q17	1207	753	584	584
   q18	7135	6265	6227	6227
   q19	1265	964	590	590
   q20	421	344	240	240
   q21	6288	2565	2673	2565
   q22	416	351	299	299
   Total cold run time: 101822 ms
   Total hot run time: 29362 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6758	6709	6803	6709
   q2	742	616	594	594
   q3	5662	5737	5311	5311
   q4	2037	2186	1574	1574
   q5	5623	5566	5781	5566
   q6	249	191	149	149
   q7	2292	2026	1828	1828
   q8	3143	2829	2777	2777
   q9	8316	8167	8204	8167
   q10	3673	3627	3264	3264
   q11	605	408	374	374
   q12	661	736	506	506
   q13	2757	3091	2389	2389
   q14	286	280	270	270
   q15	q16	681	688	605	605
   q17	7844	7014	6896	6896
   q18	13156	12317	12979	12317
   q19	919	816	836	816
   q20	2189	2200	1946	1946
   q21	5864	4705	4833	4705
   q22	491	453	427	427
   Total cold run time: 73948 ms
   Total hot run time: 67190 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6150705719

   # FE Regression Coverage Report
   Increment line coverage `  0.00% (0/14)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6184991198

   # FE Regression Coverage Report
   Increment line coverage `  100% (0/0)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6148724144

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28899 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit b3039f6f48862c5ba02ef6ab5bb486daefb9c802, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17598	5618	5532	5532
   q2	2097	339	253	253
   q3	10227	1378	827	827
   q4	4676	470	340	340
   q5	7432	785	553	553
   q6	187	184	151	151
   q7	753	788	605	605
   q8	9238	1302	1068	1068
   q9	5706	4475	4428	4428
   q10	6827	1281	1015	1015
   q11	429	250	231	231
   q12	634	395	285	285
   q13	18051	3053	2377	2377
   q14	279	266	241	241
   q15	q16	736	722	683	683
   q17	1170	728	567	567
   q18	7070	6223	6168	6168
   q19	1109	953	588	588
   q20	384	328	223	223
   q21	5534	2469	2560	2469
   q22	403	337	295	295
   Total cold run time: 100540 ms
   Total hot run time: 28899 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6455	6550	6467	6467
   q2	688	584	582	582
   q3	5561	5818	5188	5188
   q4	2355	2131	1579	1579
   q5	5758	5738	5527	5527
   q6	245	194	148	148
   q7	2354	2009	1808	1808
   q8	3118	2770	2738	2738
   q9	8332	8014	7746	7746
   q10	3654	3699	3223	3223
   q11	596	395	378	378
   q12	659	685	499	499
   q13	2741	3057	2391	2391
   q14	281	296	268	268
   q15	q16	666	678	609	609
   q17	7769	6911	6877	6877
   q18	13144	12265	13025	12265
   q19	939	787	788	787
   q20	2177	2174	1937	1937
   q21	5798	4692	4771	4692
   q22	498	452	407	407
   Total cold run time: 73788 ms
   Total hot run time: 66116 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4255623057


##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +927,42 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
         return atPin;
     }
 
+    private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+        List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+                .collect(Collectors.toList());
+        if (encodedNames.isEmpty()) {
+            return names;
+        }
+        // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+        // The storage location identifies the physical path regardless of the sync extractor's decoding
+        // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+        List<HmsPartitionInfo> partitions = hmsClient.getPartitions(

Review Comment:
   [P2] Independently confirmed at `0d0163fc17d6151c60c9cc4a711b311034bb5a93`. The new location lookup runs before partition pruning, while `CachingHmsClient` caches names and partition objects independently. A stale encoded name therefore makes a query for a surviving partition fail when the object lookup misses; the merge-base path did not perform this strict lookup. Please use `getExistingPartitions`, associate returned objects by partition identity rather than list index, and omit confirmed deleted names while retaining malformed-result errors. Add a test with a cached encoded name whose HMS partition has been deleted.



##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+    private static final int MAX_EXPANSIONS = 10000;
+    private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+    private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+    private static final Pattern BRACE_GROUP = Pattern.compile("\\{([^{}]*)\\}");

Review Comment:
   [P2] Independently confirmed at `0d0163fc17d6151c60c9cc4a711b311034bb5a93` against the merge-base normalizer. For `data_{1..2,\{special\}}.csv`, the outer group is not matched, so the numeric alternative remains literal and `data_1.csv` / `data_2.csv` are omitted. If the literal-brace alternative exists, discovery can still succeed with an incomplete file set. Please parse brace boundaries with escape awareness, reusing or extracting the existing `findClosingBrace` / `splitBraceAlternatives` logic where appropriate, and add this mixed numeric-range/literal-brace case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6143622861

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #68998:
URL: https://github.com/apache/doris/pull/68998


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6138848620

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6138853541

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4257799798


##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath, String excludedChildPat
                 }
                 deleteDir(dir);
             }
-            List<FileEntry> files = getFileSystem().listFiles(Location.of(targetPath));
+            // Storage implementations may treat listFiles arguments as globs. Overwrite must enumerate
+            // the literal target, or a table name such as table_{prod} can leave old rows behind.
+            List<FileEntry> files = new ArrayList<>();
+            try (FileIterator iterator = getFileSystem().list(Location.of(targetPath))) {
+                while (iterator.hasNext()) {
+                    FileEntry file = iterator.next();
+                    if (!file.isDirectory()) {

Review Comment:
   [P1] Keep staged files out of recursive overwrite cleanup
   
   For an unpartitioned Hive table at `abfss://workspace@onelake.dfs.fabric.microsoft.com/lake.lakehouse/Files/table_*` with `hive.staging_dir=.doris_staging`, `LocationPath` selects `FILE_HDFS`, so new files are staged beneath the target. `AzureFileSystem.list()` recursively emits those staged blobs as non-directories. The exclusion in the directory loop never protects them: this loop collects and deletes both staged input and old data before `prepareAlterTable` schedules the move, leaving the copy without its source and no restoration task for old rows. Previously the wildcard `listFiles()` branch rejected descendant keys, leaving the staging source intact even for an empty table. Exclude every file beneath `excludedChildPath`, or use literal nonrecursive enumeration, and cover this OneLake branch. This is distinct from the fixed DFS glob-matching issue in thread 4256817233. The literal-directory applicability follows [ADLS directory naming](https://learn.microsoft.com/en-us/rest/a
 pi/storageservices/naming-and-referencing-containers--blobs--and-metadata#directory-names) and the documented [OneLake API parity](https://learn.microsoft.com/en-us/fabric/onelake/onelake-api-parity); the endpoint behavior is inferred from that parity and the traced code, not a live service test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6151245566

   # FE Regression Coverage Report
   Increment line coverage `  0.00% (0/14)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256045503


##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +928,52 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
         return atPin;
     }
 
+    private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+        List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+                .collect(Collectors.toList());
+        if (encodedNames.isEmpty()) {
+            return names;
+        }
+        // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+        // The storage location identifies the physical path regardless of the sync extractor's decoding
+        // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+        // Name and object caches expire independently. A deleted encoded partition must not fail
+        // a query before pruning; surviving objects must be associated by values, not list position.
+        List<HmsPartitionInfo> partitions = hmsClient.getExistingPartitions(
+                handle.getDbName(), handle.getTableName(), encodedNames);
+        Map<List<String>, String> namesByValues = new HashMap<>();
+        for (String name : encodedNames) {
+            List<String> values = Arrays.stream(name.split("/"))
+                    .map(part -> HudiScanPlanProvider.unescapePathName(part.substring(part.indexOf('=') + 1)))
+                    .collect(Collectors.toList());
+            namesByValues.put(values, name);
+        }
+        Map<String, String> paths = new HashMap<>();
+        for (HmsPartitionInfo partition : partitions) {
+            String name = namesByValues.get(partition.getValues());
+            paths.put(name, relativePartitionPath(handle.getBasePath(), partition.getLocation()));
+        }
+        return names.stream().filter(name -> name.indexOf('%') < 0 || paths.containsKey(name))
+                .map(name -> paths.getOrDefault(name, name)).collect(Collectors.toList());
+    }
+
+    private static String relativePartitionPath(String basePath, String location) {
+        if (location == null || location.isEmpty()) {
+            throw new DorisConnectorException("Hudi HMS partition has no storage location");
+        }
+        // Path quotes literal percent signs when creating its URI; getPath retains the Hudi path's
+        // own escapes. URI.create(location).getPath() would incorrectly consume that escaping layer.
+        URI base = new Path(basePath).toUri();
+        URI partition = new Path(location).toUri();
+        String prefix = base.getPath().endsWith("/") ? base.getPath() : base.getPath() + "/";
+        if (!Objects.equals(base.getScheme(), partition.getScheme())

Review Comment:
   [P2] Accept equivalent storage schemes in Hudi partition locations
   
   Normal Hive sync can produce a table at `s3a://b/t` and live partitions at `s3://b/t/...`: [Hudi 1.0.2 keeps an existing table when only the scheme changes](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java#L224-L235), while [new partition locations use the current sync base path](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java#L179-L193). Doris also maps both schemes to `S3AFileSystem`. With `use_hive_sync_partition=true`, an encoded partition such as `dt=a%253Ab` whose location is `s3://b/t/dt=a%3Ab` now throws "outside the table base path" against `s3a://b/t`. Because location resolution precedes pruning, even `WHERE dt='ok'` on an existing plain partition fails; the previous path pruned that HMS name without this check. Compare supported equivalent scheme identities while retaining the bucket and path containment ch
 ecks, and cover mixed-scheme live partitions in both listing and pruning tests.



##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+    private static final int MAX_EXPANSIONS = 10000;
+    private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+    private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+
+    private GlobPattern() {
+    }
+
+    /**
+     * Expands {@code {N..M}} numeric range syntax in a glob pattern to the equivalent
+     * comma-separated alternation {@code {N,N+1,...,M}} understood by Hadoop and object-storage globs.
+     * Supports simple non-negative ranges like {@code {1..3}}, reverse ranges like {@code {3..1}},
+     * and mixed comma-separated patterns like {@code {1..2,3,1..3}} or {@code {Refrain,1..3}}.
+     * For simple (no-comma) brace groups, only non-negative ranges are expanded;
+     * negative-start ranges like {@code {-1..1}} are left unchanged.
+     * Duplicate values are removed. For example:
+     * <ul>
+     *   <li>{@code data_{1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+     *   <li>{@code data_{1..2,3,1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+     *   <li>{@code data_{Refrain,1..3}.csv} → {@code data_{Refrain,1,2,3}.csv}</li>
+     *   <li>{@code data_{-1..1}.csv} → unchanged (no expansion)</li>
+     * </ul>
+     */
+    public static String expandNumericRanges(String pattern) {
+        StringBuilder result = new StringBuilder();
+        Deque<BraceGroup> groups = new ArrayDeque<>();
+        ExpansionBudget budget = new ExpansionBudget();
+        boolean characterClass = false;
+        for (int i = 0; i < pattern.length(); i++) {
+            char c = pattern.charAt(i);
+            StringBuilder output = groups.isEmpty() ? result : groups.peek().segment;
+            // Escaped braces and commas are literal characters, even inside an alternative.
+            // Character classes likewise must not change the surrounding brace structure.
+            if (c == '\\') {
+                output.append(c);
+                if (i + 1 < pattern.length()) {
+                    output.append(pattern.charAt(++i));
+                }
+            } else if (characterClass) {
+                output.append(c);
+                if (c == ']') {
+                    characterClass = false;
+                }
+            } else if (c == '[') {
+                characterClass = true;
+                output.append(c);
+            } else if (c == '{') {
+                groups.push(new BraceGroup());
+            } else if (c == ',' && !groups.isEmpty()) {
+                groups.peek().finishSegment();
+            } else if (c == '}' && !groups.isEmpty()) {
+                BraceGroup group = groups.pop();
+                group.finishSegment();
+                String expanded = group.expand(budget);
+                (groups.isEmpty() ? result : groups.peek().segment).append('{').append(expanded).append('}');
+            } else {
+                output.append(c);
+            }
+        }
+        while (!groups.isEmpty()) {
+            BraceGroup group = groups.pop();
+            group.finishSegment();
+            (groups.isEmpty() ? result : groups.peek().segment)
+                    .append('{').append(String.join(",", group.segments));
+        }
+        return result.toString();
+    }
+
+    private static final class ExpansionBudget {
+        private long combinations = 1;
+        private long expandedValues;
+    }
+
+    private static final class BraceGroup {
+        private final List<String> segments = new ArrayList<>();
+        private final StringBuilder segment = new StringBuilder();
+
+        private void finishSegment() {
+            segments.add(segment.toString());
+            segment.setLength(0);
+        }
+
+        private String expand(ExpansionBudget budget) {
+            if (segments.size() == 1 && !SIMPLE_RANGE.matcher(segments.get(0)).matches()) {
+                return segments.get(0);
+            }
+            Set<String> values = new LinkedHashSet<>();
+            boolean expanded = false;
+            for (String alternative : segments) {
+                Matcher range = RANGE.matcher(alternative.trim());
+                if (range.matches()) {
+                    // Count before materializing, and use long arithmetic so an int endpoint
+                    // cannot wrap the loop or turn a huge requested range into a small count.
+                    long from = Integer.parseInt(range.group(1));
+                    long to = Integer.parseInt(range.group(2));
+                    budget.expandedValues += Math.abs(to - from) + 1;
+                    checkLimit(budget.expandedValues);
+                    long step = from <= to ? 1 : -1;
+                    for (long value = from; step > 0 ? value <= to : value >= to; value += step) {
+                        values.add(Long.toString(value));
+                    }
+                    expanded = true;
+                } else {
+                    values.add(alternative.trim());
+                }
+            }
+            if (expanded) {
+                budget.combinations *= values.size();

Review Comment:
   [P2] Count mutually exclusive ranges as alternatives
   
   For `hdfs://namenode:8020/data_{a{1..101},b{1..101}}.csv`, the two inner ranges form 202 alternatives, but this shared product becomes `101 * 101 = 10201` and throws `IllegalArgumentException` even though only 202 numeric values are generated. Both `DFSFileSystem.listFiles` and `globListWithLimit` call this normalizer before `globStatus`, so discovery fails for a small valid nested glob. [Hadoop 3.4.2 supports nested brace alternatives](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GlobPattern.java#L102-L114). Track cardinality per brace expression, summing alternative branches and multiplying only independent concatenated groups, while retaining the allocation limit. Add a nested-alternative case alongside the existing Cartesian-product limit test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6148922408

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 151904 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit b3039f6f48862c5ba02ef6ab5bb486daefb9c802, data reload: false
   
   query5	4328	621	458	458
   query6	412	200	196	196
   query7	4804	446	228	228
   query8	310	182	165	165
   query9	8798	3954	3931	3931
   query10	451	310	251	251
   query11	5843	3436	3185	3185
   query12	146	92	86	86
   query13	1267	458	329	329
   query14	6560	4811	4477	4477
   query14_1	4280	4262	4169	4169
   query15	210	204	186	186
   query16	925	449	351	351
   query17	854	672	559	559
   query18	2417	425	310	310
   query19	199	166	135	135
   query20	81	83	78	78
   query21	207	133	112	112
   query22	13032	13040	12866	12866
   query23	13155	12564	12125	12125
   query23_1	12231	12352	12292	12292
   query24	7131	820	456	456
   query24_1	474	470	473	470
   query25	536	423	370	370
   query26	1268	262	141	141
   query27	2759	445	275	275
   query28	4574	1952	1956	1952
   query29	1540	593	464	464
   query30	295	217	183	183
   query31	902	763	645	645
   query32	144	96	91	91
   query33	503	306	249	249
   query34	882	846	514	514
   query35	759	758	678	678
   query36	796	791	740	740
   query37	128	105	90	90
   query38	1831	1733	1718	1718
   query39	721	730	667	667
   query39_1	673	663	683	663
   query40	226	118	105	105
   query41	74	70	72	70
   query42	95	90	83	83
   query43	369	375	326	326
   query44	1314	707	704	704
   query45	185	178	167	167
   query46	852	955	572	572
   query47	3055	2979	2864	2864
   query48	291	317	221	221
   query49	569	392	306	306
   query50	672	268	225	225
   query51	10708	10625	10678	10625
   query52	86	82	71	71
   query53	187	203	153	153
   query54	248	182	173	173
   query55	74	67	67	67
   query56	222	209	194	194
   query57	1613	1578	1580	1578
   query58	281	260	255	255
   query59	2343	2373	2129	2129
   query60	269	238	213	213
   query61	145	147	135	135
   query62	397	345	282	282
   query63	194	166	160	160
   query64	2717	946	769	769
   query65	3423	3384	3385	3384
   query66	1774	404	295	295
   query67	19890	20013	19851	19851
   query68	2979	993	626	626
   query69	382	283	250	250
   query70	939	839	807	807
   query71	290	226	206	206
   query72	2697	2500	2223	2223
   query73	516	518	297	297
   query74	4607	4471	4267	4267
   query75	2327	2439	1930	1930
   query76	2307	1014	606	606
   query77	356	393	301	301
   query78	9016	8977	8499	8499
   query79	977	895	515	515
   query80	533	431	334	334
   query81	527	320	275	275
   query82	577	136	114	114
   query83	302	215	174	174
   query84	306	119	99	99
   query85	790	423	357	357
   query86	317	235	230	230
   query87	1994	1929	1802	1802
   query88	3609	2710	2685	2685
   query89	344	296	260	260
   query90	1903	183	184	183
   query91	167	149	126	126
   query92	92	87	88	87
   query93	1018	995	578	578
   query94	446	303	271	271
   query95	583	405	320	320
   query96	661	553	231	231
   query97	2468	2433	2331	2331
   query98	154	147	143	143
   query99	802	741	643	643
   Total cold run time: 233487 ms
   Total hot run time: 151904 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6140435893

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6143622150

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6183261390

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.63 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit d715bf7dedee46773863d4ef767b1cde328a913d, data reload: false
   
   query1	0.00	0.00	0.01
   query2	0.10	0.05	0.04
   query3	0.26	0.12	0.11
   query4	1.61	0.13	0.12
   query5	0.25	0.25	0.24
   query6	1.17	0.69	0.68
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.40	0.35	0.35
   query10	0.57	0.56	0.58
   query11	0.20	0.14	0.13
   query12	0.18	0.14	0.14
   query13	0.50	0.51	0.49
   query14	1.04	1.01	1.01
   query15	0.63	0.61	0.61
   query16	0.34	0.34	0.35
   query17	1.14	1.20	1.14
   query18	0.23	0.21	0.20
   query19	2.08	1.97	2.01
   query20	0.02	0.01	0.01
   query21	15.49	0.26	0.14
   query22	4.45	0.05	0.05
   query23	16.25	0.32	0.13
   query24	2.93	0.57	0.36
   query25	0.13	0.05	0.05
   query26	1.04	0.24	0.16
   query27	0.05	0.03	0.04
   query28	1.92	0.64	0.41
   query29	12.50	4.20	3.35
   query30	0.28	0.15	0.13
   query31	2.78	0.57	0.34
   query32	3.23	0.62	0.52
   query33	3.16	3.26	3.33
   query34	15.55	4.00	3.41
   query35	3.40	3.34	3.32
   query36	0.57	0.43	0.44
   query37	0.09	0.07	0.06
   query38	0.05	0.04	0.03
   query39	0.04	0.03	0.03
   query40	0.18	0.15	0.14
   query41	0.08	0.03	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.03	0.03
   Total cold run time: 95.06 s
   Total hot run time: 24.63 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6139588509

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.41 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 0d0163fc17d6151c60c9cc4a711b311034bb5a93, data reload: false
   
   query1	0.01	0.01	0.01
   query2	0.09	0.05	0.04
   query3	0.27	0.12	0.10
   query4	1.61	0.12	0.12
   query5	0.27	0.24	0.24
   query6	1.20	0.68	0.70
   query7	0.04	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.42	0.35	0.35
   query10	0.56	0.56	0.56
   query11	0.20	0.15	0.14
   query12	0.18	0.14	0.14
   query13	0.48	0.48	0.48
   query14	0.98	0.96	0.97
   query15	0.64	0.61	0.60
   query16	0.35	0.35	0.34
   query17	1.13	1.11	1.15
   query18	0.23	0.21	0.21
   query19	2.11	1.97	2.01
   query20	0.02	0.01	0.02
   query21	15.49	0.28	0.14
   query22	4.50	0.05	0.05
   query23	16.27	0.33	0.12
   query24	3.19	0.56	0.42
   query25	0.13	0.07	0.04
   query26	1.33	0.25	0.15
   query27	0.04	0.05	0.04
   query28	2.04	0.62	0.40
   query29	12.51	4.22	3.28
   query30	0.29	0.15	0.15
   query31	2.77	0.59	0.35
   query32	3.23	0.62	0.53
   query33	3.16	3.17	3.31
   query34	15.58	4.01	3.38
   query35	3.38	3.31	3.36
   query36	0.56	0.43	0.42
   query37	0.10	0.07	0.06
   query38	0.05	0.05	0.04
   query39	0.03	0.02	0.03
   query40	0.17	0.14	0.14
   query41	0.08	0.03	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.04	0.03
   Total cold run time: 95.82 s
   Total hot run time: 24.41 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141992016

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.56 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 2cfee92005dbc607c8ac31897868be045670ef6f, data reload: false
   
   query1	0.01	0.01	0.01
   query2	0.10	0.05	0.04
   query3	0.27	0.12	0.12
   query4	1.62	0.13	0.12
   query5	0.28	0.25	0.26
   query6	1.16	0.70	0.69
   query7	0.04	0.01	0.00
   query8	0.06	0.04	0.04
   query9	0.41	0.36	0.35
   query10	0.60	0.61	0.57
   query11	0.20	0.14	0.13
   query12	0.18	0.15	0.14
   query13	0.50	0.50	0.50
   query14	0.98	0.97	0.95
   query15	0.65	0.62	0.59
   query16	0.35	0.34	0.34
   query17	1.14	1.10	1.14
   query18	0.22	0.20	0.21
   query19	2.12	2.03	1.96
   query20	0.01	0.01	0.01
   query21	15.50	0.27	0.15
   query22	4.41	0.06	0.05
   query23	16.24	0.34	0.12
   query24	3.11	0.56	0.39
   query25	0.12	0.05	0.06
   query26	1.03	0.25	0.14
   query27	0.05	0.05	0.04
   query28	1.92	0.62	0.39
   query29	12.57	4.23	3.33
   query30	0.28	0.14	0.13
   query31	2.77	0.58	0.34
   query32	3.23	0.62	0.51
   query33	3.22	3.25	3.30
   query34	15.42	3.96	3.39
   query35	3.32	3.36	3.36
   query36	0.56	0.47	0.49
   query37	0.09	0.06	0.06
   query38	0.05	0.04	0.04
   query39	0.04	0.03	0.02
   query40	0.17	0.15	0.14
   query41	0.09	0.03	0.02
   query42	0.04	0.03	0.03
   query43	0.04	0.03	0.04
   Total cold run time: 95.17 s
   Total hot run time: 24.56 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141852946

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6144503486

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28793 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 77483a190e1adec4a2b0a53339d2cceeddbe262b, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17571	5658	5530	5530
   q2	2102	302	242	242
   q3	10242	1386	826	826
   q4	4681	475	366	366
   q5	7425	794	537	537
   q6	188	199	151	151
   q7	780	811	612	612
   q8	9252	1338	1078	1078
   q9	5690	4463	4411	4411
   q10	6816	1309	1025	1025
   q11	423	254	251	251
   q12	634	409	288	288
   q13	18059	3098	2379	2379
   q14	279	270	245	245
   q15	q16	742	732	663	663
   q17	1202	745	570	570
   q18	7029	6215	6167	6167
   q19	1122	953	626	626
   q20	375	338	227	227
   q21	5525	2329	2358	2329
   q22	360	311	270	270
   Total cold run time: 100497 ms
   Total hot run time: 28793 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6121	6132	6104	6104
   q2	670	587	564	564
   q3	5222	5369	4931	4931
   q4	2017	2131	1570	1570
   q5	5250	5187	5893	5187
   q6	250	205	157	157
   q7	2291	2082	1898	1898
   q8	3186	2781	2775	2775
   q9	8246	8197	8234	8197
   q10	3965	3785	3359	3359
   q11	679	465	391	391
   q12	690	743	548	548
   q13	2978	3223	2540	2540
   q14	340	335	294	294
   q15	q16	704	755	668	668
   q17	8303	7617	7469	7469
   q18	13200	12274	13079	12274
   q19	920	808	795	795
   q20	2187	2174	1951	1951
   q21	5781	4737	4818	4737
   q22	503	473	393	393
   Total cold run time: 73503 ms
   Total hot run time: 66802 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6144665063

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 152237 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 77483a190e1adec4a2b0a53339d2cceeddbe262b, data reload: false
   
   query5	4298	584	446	446
   query6	426	231	186	186
   query7	4801	442	224	224
   query8	339	179	166	166
   query9	8756	4023	4057	4023
   query10	494	319	254	254
   query11	6037	3424	3186	3186
   query12	144	91	85	85
   query13	1259	438	322	322
   query14	6551	4863	4494	4494
   query14_1	4290	4309	4249	4249
   query15	206	203	176	176
   query16	939	459	426	426
   query17	876	659	552	552
   query18	2425	433	324	324
   query19	189	170	160	160
   query20	81	80	79	79
   query21	206	140	115	115
   query22	13027	13198	12896	12896
   query23	13189	12533	12239	12239
   query23_1	12169	12155	12261	12155
   query24	7090	828	469	469
   query24_1	475	476	480	476
   query25	532	427	377	377
   query26	1257	263	139	139
   query27	2763	444	277	277
   query28	4588	1955	1987	1955
   query29	1555	584	439	439
   query30	289	211	187	187
   query31	872	760	643	643
   query32	147	103	93	93
   query33	516	295	236	236
   query34	930	850	488	488
   query35	748	766	644	644
   query36	841	832	752	752
   query37	135	102	92	92
   query38	1797	1747	1685	1685
   query39	712	721	697	697
   query39_1	664	669	700	669
   query40	213	115	92	92
   query41	67	66	65	65
   query42	86	85	81	81
   query43	362	381	328	328
   query44	1325	701	687	687
   query45	181	173	168	168
   query46	850	1025	580	580
   query47	3013	2954	2885	2885
   query48	296	312	216	216
   query49	570	402	281	281
   query50	676	283	212	212
   query51	10744	10350	10379	10350
   query52	76	83	71	71
   query53	186	208	149	149
   query54	236	189	173	173
   query55	74	77	63	63
   query56	231	231	206	206
   query57	1618	1615	1576	1576
   query58	284	288	249	249
   query59	2365	2401	2124	2124
   query60	283	238	222	222
   query61	147	151	142	142
   query62	395	364	295	295
   query63	204	163	157	157
   query64	2731	936	781	781
   query65	3437	3372	3367	3367
   query66	1773	417	307	307
   query67	20074	20109	19949	19949
   query68	2897	991	637	637
   query69	383	291	251	251
   query70	955	879	817	817
   query71	295	233	218	218
   query72	2740	2570	2270	2270
   query73	515	546	299	299
   query74	4574	4471	4330	4330
   query75	2335	2294	1972	1972
   query76	2289	1034	597	597
   query77	355	407	310	310
   query78	8915	8998	8492	8492
   query79	858	889	528	528
   query80	479	434	365	365
   query81	526	327	271	271
   query82	209	135	115	115
   query83	195	196	179	179
   query84	292	112	105	105
   query85	779	431	392	392
   query86	255	242	226	226
   query87	1975	1951	1819	1819
   query88	3601	2724	2672	2672
   query89	328	297	257	257
   query90	2098	180	181	180
   query91	158	154	122	122
   query92	99	84	93	84
   query93	992	1008	572	572
   query94	453	285	287	285
   query95	582	401	323	323
   query96	667	521	229	229
   query97	2443	2395	2361	2361
   query98	161	158	150	150
   query99	761	775	652	652
   Total cold run time: 232896 ms
   Total hot run time: 152237 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6150510724

   # FE Regression Coverage Report
   Increment line coverage `  0.00% (0/14)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6151948565

   # FE Regression Coverage Report
   Increment line coverage `  0.00% (0/14)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6139521013

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 151418 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 0d0163fc17d6151c60c9cc4a711b311034bb5a93, data reload: false
   
   query5	4330	580	449	449
   query6	422	213	203	203
   query7	4802	440	227	227
   query8	312	181	170	170
   query9	8743	3994	3994	3994
   query10	453	311	251	251
   query11	5885	3451	3199	3199
   query12	157	88	85	85
   query13	1283	450	326	326
   query14	6530	4784	4460	4460
   query14_1	4229	4258	4225	4225
   query15	208	211	182	182
   query16	935	445	423	423
   query17	874	645	539	539
   query18	2416	422	308	308
   query19	201	163	135	135
   query20	81	82	82	82
   query21	210	137	113	113
   query22	12995	13023	12786	12786
   query23	13253	12618	12080	12080
   query23_1	12269	12101	12211	12101
   query24	7115	858	496	496
   query24_1	466	456	467	456
   query25	508	399	332	332
   query26	1258	252	134	134
   query27	2771	435	268	268
   query28	4628	1965	1953	1953
   query29	1553	574	424	424
   query30	288	213	183	183
   query31	888	755	655	655
   query32	141	116	89	89
   query33	518	293	242	242
   query34	921	845	491	491
   query35	735	743	651	651
   query36	829	818	726	726
   query37	125	97	91	91
   query38	1808	1740	1682	1682
   query39	706	719	683	683
   query39_1	675	681	674	674
   query40	212	119	96	96
   query41	67	63	62	62
   query42	92	89	79	79
   query43	367	371	318	318
   query44	1321	697	698	697
   query45	178	192	165	165
   query46	836	973	563	563
   query47	2966	2944	2939	2939
   query48	299	302	221	221
   query49	571	394	321	321
   query50	667	267	208	208
   query51	10319	10532	10311	10311
   query52	82	81	74	74
   query53	181	208	150	150
   query54	245	189	182	182
   query55	75	66	63	63
   query56	240	207	204	204
   query57	1636	1598	1545	1545
   query58	287	255	257	255
   query59	2375	2399	2125	2125
   query60	276	233	214	214
   query61	146	148	140	140
   query62	388	339	284	284
   query63	193	161	178	161
   query64	2735	908	773	773
   query65	3436	3349	3378	3349
   query66	1779	428	321	321
   query67	20301	20298	19889	19889
   query68	3190	987	628	628
   query69	379	303	251	251
   query70	935	854	816	816
   query71	294	228	211	211
   query72	2786	2612	2309	2309
   query73	520	533	292	292
   query74	4602	4445	4266	4266
   query75	2335	2293	1929	1929
   query76	2375	1003	620	620
   query77	353	393	301	301
   query78	9117	8914	8395	8395
   query79	877	837	516	516
   query80	473	409	347	347
   query81	526	327	277	277
   query82	213	135	105	105
   query83	190	197	180	180
   query84	290	120	108	108
   query85	759	431	366	366
   query86	257	239	235	235
   query87	2000	1961	1842	1842
   query88	3604	2726	2653	2653
   query89	326	298	257	257
   query90	2101	189	182	182
   query91	153	146	125	125
   query92	99	87	85	85
   query93	978	981	570	570
   query94	434	302	293	293
   query95	594	343	317	317
   query96	674	519	232	232
   query97	2461	2424	2332	2332
   query98	160	151	152	151
   query99	770	757	646	646
   Total cold run time: 233411 ms
   Total hot run time: 151418 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141853432

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6138850813

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6147701031

   # FE Regression Coverage Report
   Increment line coverage `  100% (0/0)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6144746627

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.53 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 77483a190e1adec4a2b0a53339d2cceeddbe262b, data reload: false
   
   query1	0.01	0.01	0.01
   query2	0.10	0.05	0.05
   query3	0.26	0.12	0.11
   query4	1.61	0.13	0.13
   query5	0.26	0.25	0.23
   query6	1.18	0.69	0.67
   query7	0.04	0.01	0.01
   query8	0.05	0.03	0.03
   query9	0.42	0.36	0.37
   query10	0.58	0.58	0.59
   query11	0.22	0.15	0.14
   query12	0.18	0.14	0.15
   query13	0.51	0.50	0.51
   query14	0.98	0.96	0.97
   query15	0.65	0.62	0.61
   query16	0.32	0.32	0.36
   query17	1.16	1.16	1.15
   query18	0.23	0.21	0.20
   query19	2.04	1.97	2.04
   query20	0.02	0.01	0.01
   query21	15.49	0.27	0.14
   query22	4.45	0.05	0.05
   query23	16.27	0.33	0.12
   query24	3.10	0.65	0.43
   query25	0.12	0.04	0.04
   query26	1.64	0.24	0.14
   query27	0.05	0.04	0.04
   query28	2.56	0.61	0.42
   query29	12.51	4.23	3.29
   query30	0.28	0.13	0.12
   query31	2.77	0.60	0.34
   query32	3.25	0.63	0.53
   query33	3.20	3.22	3.17
   query34	15.43	4.07	3.38
   query35	3.36	3.36	3.36
   query36	0.56	0.44	0.43
   query37	0.09	0.06	0.06
   query38	0.04	0.04	0.03
   query39	0.04	0.02	0.02
   query40	0.18	0.15	0.15
   query41	0.09	0.03	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.03	0.03
   Total cold run time: 96.38 s
   Total hot run time: 24.53 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4278222805


##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath, String excludedChildPat
                 }
                 deleteDir(dir);
             }
-            List<FileEntry> files = getFileSystem().listFiles(Location.of(targetPath));
+            // Storage implementations may treat listFiles arguments as globs. Overwrite must enumerate
+            // the literal target, or a table name such as table_{prod} can leave old rows behind.
+            List<FileEntry> files = new ArrayList<>();
+            try (FileIterator iterator = getFileSystem().list(Location.of(targetPath))) {
+                while (iterator.hasNext()) {
+                    FileEntry file = iterator.next();
+                    if (!file.isDirectory()) {

Review Comment:
   Fixed in d715bf7dede. Both cleanup loops now exclude the staging root and every descendant, using the existing filesystem-aware equality and slash-boundary containment helpers. The directory loop also excludes nested staging directories if a provider lists them directly. Literal target enumeration and existing glob discovery behavior remain unchanged.
   
   Added a transaction regression using real AzureFileSystem enumeration and copy-then-delete rename with an in-memory paginated AzureObjStorage. Before the fix it fails with FileNotFoundException for the staged source, including an otherwise empty table. The test covers 12 combinations of literal target names, empty/nonempty old data and directory-listing shapes. It checks that new data is published, other-query staging and adjacent tables survive, and direct/nested old files plus .doris_staging_backup are cleaned up. No live OneLake endpoint was used.
   
   Validation: 604 scoped Hive/Azure/HDFS/shared-HMS tests passed with zero failures, errors or skips; FE Checkstyle passed in 21 reactor modules.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6183205515

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 153402 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d715bf7dedee46773863d4ef767b1cde328a913d, data reload: false
   
   query5	4306	593	474	474
   query6	426	210	195	195
   query7	4810	450	231	231
   query8	319	178	172	172
   query9	8779	4059	4071	4059
   query10	499	320	269	269
   query11	6076	3777	3553	3553
   query12	147	91	86	86
   query13	1249	445	326	326
   query14	6100	4067	3787	3787
   query14_1	3568	3588	3569	3569
   query15	206	200	188	188
   query16	954	472	431	431
   query17	852	661	549	549
   query18	2421	445	313	313
   query19	213	159	137	137
   query20	86	92	82	82
   query21	208	143	115	115
   query22	13000	12998	12972	12972
   query23	13518	12810	12461	12461
   query23_1	12670	12847	12797	12797
   query24	7095	849	481	481
   query24_1	494	486	481	481
   query25	539	432	367	367
   query26	1366	267	138	138
   query27	2765	435	268	268
   query28	4612	1939	1943	1939
   query29	1564	573	438	438
   query30	310	206	184	184
   query31	1008	935	807	807
   query32	137	90	83	83
   query33	505	295	241	241
   query34	893	852	504	504
   query35	744	750	667	667
   query36	817	790	717	717
   query37	125	101	89	89
   query38	1814	1721	1731	1721
   query39	795	796	748	748
   query39_1	740	728	745	728
   query40	215	114	94	94
   query41	65	62	66	62
   query42	98	85	82	82
   query43	369	392	334	334
   query44	1320	696	687	687
   query45	180	174	162	162
   query46	853	981	567	567
   query47	3007	3020	2795	2795
   query48	316	345	219	219
   query49	576	398	301	301
   query50	710	288	214	214
   query51	10483	10429	10535	10429
   query52	79	96	74	74
   query53	194	205	151	151
   query54	228	189	174	174
   query55	69	64	64	64
   query56	232	213	218	213
   query57	1614	1597	1565	1565
   query58	286	265	258	258
   query59	2083	2188	1955	1955
   query60	283	241	224	224
   query61	145	146	143	143
   query62	399	340	284	284
   query63	201	162	171	162
   query64	2676	909	786	786
   query65	3341	3255	3297	3255
   query66	1772	411	302	302
   query67	20336	20077	20104	20077
   query68	2954	1043	620	620
   query69	380	287	241	241
   query70	847	809	780	780
   query71	302	244	241	241
   query72	2701	2759	2468	2468
   query73	540	566	305	305
   query74	4607	4483	4322	4322
   query75	2331	2303	1963	1963
   query76	2308	1037	630	630
   query77	354	383	295	295
   query78	9683	9701	9179	9179
   query79	1028	911	528	528
   query80	549	456	373	373
   query81	522	326	276	276
   query82	592	147	117	117
   query83	318	209	242	209
   query84	315	125	97	97
   query85	802	416	373	373
   query86	309	239	233	233
   query87	2011	1962	1827	1827
   query88	3664	2726	2710	2710
   query89	340	296	261	261
   query90	1909	192	184	184
   query91	156	150	124	124
   query92	99	90	91	90
   query93	1064	1033	578	578
   query94	453	319	275	275
   query95	596	339	383	339
   query96	675	547	234	234
   query97	2455	2436	2341	2341
   query98	161	153	165	153
   query99	778	761	642	642
   Total cold run time: 234492 ms
   Total hot run time: 153402 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4278181403


##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath, String excludedChildPat
                 }
                 deleteDir(dir);
             }
-            List<FileEntry> files = getFileSystem().listFiles(Location.of(targetPath));
+            // Storage implementations may treat listFiles arguments as globs. Overwrite must enumerate
+            // the literal target, or a table name such as table_{prod} can leave old rows behind.
+            List<FileEntry> files = new ArrayList<>();
+            try (FileIterator iterator = getFileSystem().list(Location.of(targetPath))) {
+                while (iterator.hasNext()) {
+                    FileEntry file = iterator.next();
+                    if (!file.isDirectory()) {

Review Comment:
   [P1] Independently confirmed by tracing the code at `b3039f6f48862c5ba02ef6ab5bb486daefb9c802`. This does not require an upgrade or metadata from an older master build: a fresh deployment with a newly created unpartitioned Hive table at the OneLake path above and relative `hive.staging_dir=.doris_staging` can reach the same path.
   
   `prepareAlterTable` calls this cleanup before `asyncRenameFiles`. Unlike the immediate-child DFS listing covered by the new test, `AzureFileSystem.list()` returns descendant blobs, including `.doris_staging/.../new.parquet`, with `isDirectory=false`. The directory-loop exclusion therefore does not protect the staged source; the file loop deletes both it and existing table files. The subsequent move fails, and this relative-staging branch has no old-data restoration task.
   
   Please exclude the staging root and all its descendants from the file deletion set, using path-boundary-aware comparisons such as the existing `pathsEqual` / `isSubDirectory` helpers. Add a transaction regression using the real `AzureFileSystem` with a recording object store, verifying that staged inputs survive cleanup and the overwrite commits successfully. This conclusion is source-based; I have not run a live OneLake test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6183090405

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29201 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d715bf7dedee46773863d4ef767b1cde328a913d, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17628	5653	5550	5550
   q2	2084	319	271	271
   q3	10253	1400	864	864
   q4	4685	471	345	345
   q5	7447	801	531	531
   q6	186	186	158	158
   q7	760	800	606	606
   q8	9251	1380	1077	1077
   q9	5536	4266	4252	4252
   q10	6822	1309	1019	1019
   q11	429	258	236	236
   q12	631	409	284	284
   q13	18087	3633	2948	2948
   q14	284	268	247	247
   q15	q16	739	732	670	670
   q17	1168	751	572	572
   q18	7080	6239	6188	6188
   q19	1108	951	617	617
   q20	375	332	228	228
   q21	4974	2254	2382	2254
   q22	360	312	284	284
   Total cold run time: 99887 ms
   Total hot run time: 29201 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6198	6267	6340	6267
   q2	712	591	555	555
   q3	5214	5408	4972	4972
   q4	2040	2139	1590	1590
   q5	5300	5456	5790	5456
   q6	260	200	164	164
   q7	2379	2024	1943	1943
   q8	3112	2846	2803	2803
   q9	8201	8050	8193	8050
   q10	3814	3708	3324	3324
   q11	664	469	404	404
   q12	697	740	558	558
   q13	3517	3939	3310	3310
   q14	310	334	288	288
   q15	q16	711	742	665	665
   q17	8473	7543	6960	6960
   q18	13217	12266	13092	12266
   q19	952	829	803	803
   q20	2191	2183	1935	1935
   q21	5891	4792	4969	4792
   q22	540	479	396	396
   Total cold run time: 74393 ms
   Total hot run time: 67501 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6184844864

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 77.07% (35621/46217) |
   | Line Coverage     | 62.40% (403698/646949) |
   | Region Coverage   | 58.83% (341102/579781) |
   | Branch Coverage   | 59.73% (157107/263010) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6182627946

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6146574981

   # FE Regression Coverage Report
   Increment line coverage `  100% (0/0)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6139398135

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29129 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 0d0163fc17d6151c60c9cc4a711b311034bb5a93, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17622	5624	5540	5540
   q2	2076	294	237	237
   q3	10288	1406	847	847
   q4	4679	478	344	344
   q5	7433	799	533	533
   q6	187	187	153	153
   q7	756	793	615	615
   q8	9260	1330	1053	1053
   q9	5647	4460	4452	4452
   q10	6804	1308	1027	1027
   q11	430	261	240	240
   q12	633	408	285	285
   q13	18048	3063	2383	2383
   q14	282	273	250	250
   q15	q16	748	731	680	680
   q17	1304	752	579	579
   q18	7189	6247	6214	6214
   q19	1112	957	626	626
   q20	390	336	231	231
   q21	5857	2541	2576	2541
   q22	397	356	299	299
   Total cold run time: 101142 ms
   Total hot run time: 29129 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6664	6639	6722	6639
   q2	696	582	575	575
   q3	5657	5707	5268	5268
   q4	2035	2131	1586	1586
   q5	5848	5694	5946	5694
   q6	245	185	143	143
   q7	2248	2056	1852	1852
   q8	3089	2819	2678	2678
   q9	8182	8149	7798	7798
   q10	3664	3585	3285	3285
   q11	586	399	365	365
   q12	659	706	490	490
   q13	2755	3052	2396	2396
   q14	283	294	271	271
   q15	q16	657	687	613	613
   q17	7828	7246	6905	6905
   q18	13096	12272	13095	12272
   q19	904	799	799	799
   q20	2199	2188	1939	1939
   q21	5768	4702	4821	4702
   q22	510	472	392	392
   Total cold run time: 73573 ms
   Total hot run time: 66662 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256354409


##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +928,52 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
         return atPin;
     }
 
+    private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+        List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+                .collect(Collectors.toList());
+        if (encodedNames.isEmpty()) {
+            return names;
+        }
+        // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+        // The storage location identifies the physical path regardless of the sync extractor's decoding
+        // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+        // Name and object caches expire independently. A deleted encoded partition must not fail
+        // a query before pruning; surviving objects must be associated by values, not list position.
+        List<HmsPartitionInfo> partitions = hmsClient.getExistingPartitions(
+                handle.getDbName(), handle.getTableName(), encodedNames);
+        Map<List<String>, String> namesByValues = new HashMap<>();
+        for (String name : encodedNames) {
+            List<String> values = Arrays.stream(name.split("/"))
+                    .map(part -> HudiScanPlanProvider.unescapePathName(part.substring(part.indexOf('=') + 1)))
+                    .collect(Collectors.toList());
+            namesByValues.put(values, name);
+        }
+        Map<String, String> paths = new HashMap<>();
+        for (HmsPartitionInfo partition : partitions) {
+            String name = namesByValues.get(partition.getValues());
+            paths.put(name, relativePartitionPath(handle.getBasePath(), partition.getLocation()));
+        }
+        return names.stream().filter(name -> name.indexOf('%') < 0 || paths.containsKey(name))
+                .map(name -> paths.getOrDefault(name, name)).collect(Collectors.toList());
+    }
+
+    private static String relativePartitionPath(String basePath, String location) {
+        if (location == null || location.isEmpty()) {
+            throw new DorisConnectorException("Hudi HMS partition has no storage location");
+        }
+        // Path quotes literal percent signs when creating its URI; getPath retains the Hudi path's
+        // own escapes. URI.create(location).getPath() would incorrectly consume that escaping layer.
+        URI base = new Path(basePath).toUri();
+        URI partition = new Path(location).toUri();
+        String prefix = base.getPath().endsWith("/") ? base.getPath() : base.getPath() + "/";
+        if (!Objects.equals(base.getScheme(), partition.getScheme())

Review Comment:
   [P2] Independently confirmed at `2cfee92005dbc607c8ac31897868be045670ef6f`. `getTableHandle` takes the base path from HMS, and both partition listing and `applyFilter` resolve encoded partition locations before pruning. A table at `s3a://b/t` with a live partition at `s3://b/t/dt=a%3Ab` therefore fails this check, including queries targeting another plain partition. This can arise from normal Hive sync: table-location comparison ignores the scheme, while new partitions use the current sync base path. Doris also maps `fs.s3.impl` and `fs.s3a.impl` to `S3AFileSystem`. Please compare supported equivalent storage schemes here, retaining authority/bucket and directory containment checks, and add mixed-scheme listing and pruning tests in both directions.



##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+    private static final int MAX_EXPANSIONS = 10000;
+    private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+    private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+
+    private GlobPattern() {
+    }
+
+    /**
+     * Expands {@code {N..M}} numeric range syntax in a glob pattern to the equivalent
+     * comma-separated alternation {@code {N,N+1,...,M}} understood by Hadoop and object-storage globs.
+     * Supports simple non-negative ranges like {@code {1..3}}, reverse ranges like {@code {3..1}},
+     * and mixed comma-separated patterns like {@code {1..2,3,1..3}} or {@code {Refrain,1..3}}.
+     * For simple (no-comma) brace groups, only non-negative ranges are expanded;
+     * negative-start ranges like {@code {-1..1}} are left unchanged.
+     * Duplicate values are removed. For example:
+     * <ul>
+     *   <li>{@code data_{1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+     *   <li>{@code data_{1..2,3,1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+     *   <li>{@code data_{Refrain,1..3}.csv} → {@code data_{Refrain,1,2,3}.csv}</li>
+     *   <li>{@code data_{-1..1}.csv} → unchanged (no expansion)</li>
+     * </ul>
+     */
+    public static String expandNumericRanges(String pattern) {
+        StringBuilder result = new StringBuilder();
+        Deque<BraceGroup> groups = new ArrayDeque<>();
+        ExpansionBudget budget = new ExpansionBudget();
+        boolean characterClass = false;
+        for (int i = 0; i < pattern.length(); i++) {
+            char c = pattern.charAt(i);
+            StringBuilder output = groups.isEmpty() ? result : groups.peek().segment;
+            // Escaped braces and commas are literal characters, even inside an alternative.
+            // Character classes likewise must not change the surrounding brace structure.
+            if (c == '\\') {
+                output.append(c);
+                if (i + 1 < pattern.length()) {
+                    output.append(pattern.charAt(++i));
+                }
+            } else if (characterClass) {
+                output.append(c);
+                if (c == ']') {
+                    characterClass = false;
+                }
+            } else if (c == '[') {
+                characterClass = true;
+                output.append(c);
+            } else if (c == '{') {
+                groups.push(new BraceGroup());
+            } else if (c == ',' && !groups.isEmpty()) {
+                groups.peek().finishSegment();
+            } else if (c == '}' && !groups.isEmpty()) {
+                BraceGroup group = groups.pop();
+                group.finishSegment();
+                String expanded = group.expand(budget);
+                (groups.isEmpty() ? result : groups.peek().segment).append('{').append(expanded).append('}');
+            } else {
+                output.append(c);
+            }
+        }
+        while (!groups.isEmpty()) {
+            BraceGroup group = groups.pop();
+            group.finishSegment();
+            (groups.isEmpty() ? result : groups.peek().segment)
+                    .append('{').append(String.join(",", group.segments));
+        }
+        return result.toString();
+    }
+
+    private static final class ExpansionBudget {
+        private long combinations = 1;
+        private long expandedValues;
+    }
+
+    private static final class BraceGroup {
+        private final List<String> segments = new ArrayList<>();
+        private final StringBuilder segment = new StringBuilder();
+
+        private void finishSegment() {
+            segments.add(segment.toString());
+            segment.setLength(0);
+        }
+
+        private String expand(ExpansionBudget budget) {
+            if (segments.size() == 1 && !SIMPLE_RANGE.matcher(segments.get(0)).matches()) {
+                return segments.get(0);
+            }
+            Set<String> values = new LinkedHashSet<>();
+            boolean expanded = false;
+            for (String alternative : segments) {
+                Matcher range = RANGE.matcher(alternative.trim());
+                if (range.matches()) {
+                    // Count before materializing, and use long arithmetic so an int endpoint
+                    // cannot wrap the loop or turn a huge requested range into a small count.
+                    long from = Integer.parseInt(range.group(1));
+                    long to = Integer.parseInt(range.group(2));
+                    budget.expandedValues += Math.abs(to - from) + 1;
+                    checkLimit(budget.expandedValues);
+                    long step = from <= to ? 1 : -1;
+                    for (long value = from; step > 0 ? value <= to : value >= to; value += step) {
+                        values.add(Long.toString(value));
+                    }
+                    expanded = true;
+                } else {
+                    values.add(alternative.trim());
+                }
+            }
+            if (expanded) {
+                budget.combinations *= values.size();

Review Comment:
   [P2] Independently confirmed at `2cfee92005dbc607c8ac31897868be045670ef6f`. For `data_{a{1..101},b{1..101}}.csv`, the two inner groups share this budget: closing the first sets `combinations=101`, and closing the second sets it to `10201`, although the branches are mutually exclusive and produce only 202 candidates (`expandedValues` is also 202). Both DFS discovery entry points fail before Hadoop sees the pattern. Please track candidate counts per expression, summing alternative branches and multiplying independent concatenated groups, while retaining the global allocation limit. Add this 202-candidate success case and keep the `{1..101}/{1..101}` Cartesian-product rejection test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141911686

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 152533 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 2cfee92005dbc607c8ac31897868be045670ef6f, data reload: false
   
   query5	4320	608	457	457
   query6	426	222	183	183
   query7	4805	449	220	220
   query8	319	183	164	164
   query9	8782	4013	3941	3941
   query10	474	302	260	260
   query11	5996	3420	3172	3172
   query12	145	86	84	84
   query13	1253	450	309	309
   query14	6520	4796	4492	4492
   query14_1	4308	4301	4244	4244
   query15	199	196	180	180
   query16	942	422	400	400
   query17	845	628	531	531
   query18	2402	411	288	288
   query19	194	156	124	124
   query20	78	79	76	76
   query21	210	136	121	121
   query22	12972	13070	12859	12859
   query23	13252	12453	12271	12271
   query23_1	12257	12332	12340	12332
   query24	7072	800	451	451
   query24_1	471	469	487	469
   query25	507	395	334	334
   query26	1254	255	134	134
   query27	2775	439	268	268
   query28	4629	1937	1941	1937
   query29	1510	579	439	439
   query30	330	218	186	186
   query31	877	771	645	645
   query32	152	108	82	82
   query33	505	297	230	230
   query34	918	847	487	487
   query35	735	746	641	641
   query36	834	840	739	739
   query37	138	101	90	90
   query38	1781	1764	1678	1678
   query39	721	707	658	658
   query39_1	669	693	656	656
   query40	218	113	92	92
   query41	64	63	65	63
   query42	84	85	82	82
   query43	362	376	310	310
   query44	1316	699	698	698
   query45	184	192	163	163
   query46	858	961	564	564
   query47	3039	3009	2940	2940
   query48	289	302	216	216
   query49	562	407	287	287
   query50	675	281	210	210
   query51	10673	10510	10603	10510
   query52	77	78	68	68
   query53	183	206	151	151
   query54	229	192	208	192
   query55	69	66	65	65
   query56	238	214	193	193
   query57	1600	1576	1577	1576
   query58	272	256	246	246
   query59	2356	2392	2144	2144
   query60	269	236	223	223
   query61	150	148	152	148
   query62	394	349	291	291
   query63	197	170	171	170
   query64	2851	1035	873	873
   query65	3452	3383	3385	3383
   query66	1789	425	313	313
   query67	20319	20179	20068	20068
   query68	2980	1000	621	621
   query69	391	308	264	264
   query70	919	842	849	842
   query71	279	229	222	222
   query72	2801	2694	2336	2336
   query73	524	521	297	297
   query74	4590	4505	4303	4303
   query75	2341	2285	1965	1965
   query76	2287	1016	599	599
   query77	357	409	304	304
   query78	9173	9040	8492	8492
   query79	1030	907	519	519
   query80	534	413	337	337
   query81	529	325	269	269
   query82	788	141	116	116
   query83	306	197	177	177
   query84	303	118	104	104
   query85	788	462	369	369
   query86	317	242	228	228
   query87	2015	1984	1884	1884
   query88	3712	2740	2707	2707
   query89	338	298	253	253
   query90	1895	194	183	183
   query91	157	147	122	122
   query92	98	86	93	86
   query93	1014	999	569	569
   query94	459	287	298	287
   query95	592	346	385	346
   query96	683	521	235	235
   query97	2509	2428	2294	2294
   query98	157	145	144	144
   query99	777	782	672	672
   Total cold run time: 234796 ms
   Total hot run time: 152533 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256817233


##########
fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java:
##########
@@ -277,7 +279,7 @@ public List<FileEntry> listFiles(Location location) throws IOException {
     }
 
     private static boolean containsGlob(String path) {
-        return path.contains("*") || path.contains("?") || path.contains("[");
+        return path.contains("*") || path.contains("?") || path.contains("[") || path.contains("{");

Review Comment:
   [P1] Preserve literal directory listings when enabling brace globs
   
   For an unpartitioned Hive table at `hdfs://nn/warehouse/table_{prod}` with `hive.staging_dir=.doris_staging`, `INSERT OVERWRITE` calls `HiveConnectorTransaction.deleteTargetPathContents`, which uses `listFiles(targetPath)` to remove old files before moving staged files into the table. This new `{` check sends that literal directory through `globStatus`: [Hadoop interprets `{prod}` as a group](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GlobPattern.java#L102-L114), matching `table_prod`, and [returns an empty array when it finds no match](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L355-L377). Cleanup therefore succeeds without deleting old files, and the overwrite leaves old and new rows together. The previous direct listing read the literal directory correctly. Preserve literal-directory behavior for the
 se callers while retaining brace-glob discovery, and cover a relative-staging overwrite into a directory containing braces.



##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -794,7 +799,8 @@ private List<ConnectorPartitionInfo> collectPartitions(HudiTableHandle handle, b
                 // HoodieTableMetaClient built just to read a timeline zone - plus a fallback that
                 // would have published epoch 0 as a partition's last-modified time for any
                 // non-numeric pin, the day it became reachable.
-                return buildPartitionInfos(hmsNames, partKeyNames, latestInstantMillis(handle));
+                return buildPartitionInfos(hmsPartitionPaths(handle, hmsNames),

Review Comment:
   [P2] Preserve literal percent values in unencoded Hudi partitions
   
   With `use_hive_sync_partition=true`, [Hudi's default `hoodie.datasource.write.partitionpath.urlencode=false`](https://github.com/apache/hudi/blob/release-1.0.2/hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorOptions.java#L33-L37) permits the literal STRING value `%3A` at `s3://b/t/dt=%3A`. [The Hive-style sync extractor retains `%3A`](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveStylePartitionValueExtractor.java#L32-L39), so HMS exposes `dt=%253A`. Previously `buildPartitionInfos` decoded that HMS name once and published `%3A`; the new location substitution supplies `dt=%3A`, which `parsePartitionValues` decodes into `:`. Consequently `partition_values()` now reports the wrong logical value and `SHOW PARTITIONS` changes the name; separate raw `:` and `%3A` partitions can also collapse in name-keyed consumers. Preserve the table's encoding contract when deriving logical values from physical p
 aths. Add a raw `%3A` location case alongside the existing encoded `%253A` case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6149018551

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.61 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit b3039f6f48862c5ba02ef6ab5bb486daefb9c802, data reload: false
   
   query1	0.01	0.01	0.00
   query2	0.10	0.05	0.05
   query3	0.26	0.12	0.12
   query4	1.61	0.12	0.13
   query5	0.27	0.24	0.25
   query6	1.19	0.69	0.68
   query7	0.04	0.01	0.00
   query8	0.05	0.04	0.04
   query9	0.43	0.35	0.36
   query10	0.61	0.56	0.56
   query11	0.20	0.14	0.15
   query12	0.18	0.15	0.15
   query13	0.49	0.49	0.52
   query14	0.99	0.96	0.96
   query15	0.63	0.60	0.63
   query16	0.35	0.35	0.35
   query17	1.09	1.11	1.15
   query18	0.23	0.21	0.20
   query19	2.12	2.01	2.03
   query20	0.02	0.01	0.02
   query21	15.49	0.25	0.14
   query22	4.40	0.06	0.05
   query23	16.25	0.31	0.12
   query24	3.10	0.62	0.50
   query25	0.11	0.06	0.06
   query26	1.81	0.24	0.14
   query27	0.05	0.04	0.05
   query28	2.96	0.66	0.42
   query29	12.49	4.14	3.27
   query30	0.28	0.14	0.13
   query31	2.78	0.59	0.34
   query32	3.23	0.61	0.52
   query33	3.21	3.20	3.18
   query34	15.60	3.97	3.38
   query35	3.34	3.34	3.38
   query36	0.55	0.45	0.47
   query37	0.09	0.06	0.06
   query38	0.05	0.03	0.04
   query39	0.04	0.03	0.04
   query40	0.17	0.14	0.15
   query41	0.08	0.03	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.03	0.04
   Total cold run time: 97.03 s
   Total hot run time: 24.61 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org