You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/13 05:21:45 UTC
[PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Gabriel39 opened a new pull request, #68998:
URL: https://github.com/apache/doris/pull/68998
### What problem does this PR solve?
External connector queries can fail during catalog initialization, file listing, or partition pruning after dependency isolation. This fixes the following paths:
- Keep DLF and Paimon OSS adapters with their plugin APIs, isolate the Hive expression implementation required by DLF, and share the host Jindo SDK/native runtime. Use the DLF client's actual constructor signature, and preserve standard Hive storage-format class names in persisted metadata.
- Normalize numeric ranges for Hadoop file listing and recognize legacy OSS authorities while preserving raw object keys and the configured endpoint.
- Resolve ambiguous escaped Hudi partition names through their storage locations, keep TIMESTAMPTZ predicates for typed filtering, and pass the session timezone when BE parses offset-free partition values.
- Continue MaxCompute schema enumeration when a listed schema disappears during its lazy reload, while preserving other errors.
The QL isolation serves DLF's local expression-proxy initialization. Current connector metadata operations use partition-name APIs; private Kryo expressions are not treated as an external wire format.
### Release note
Fix external catalog initialization and file/partition discovery failures affecting DLF, Paimon OSS, Hudi, Hadoop glob paths, legacy OSS locations, and concurrent MaxCompute schema deletion.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (isolated packaged plugins with a shared Jindo runtime)
- Behavior changed:
- [x] Yes. Previously failing connector paths now preserve the expected catalog, file, and partition semantics. Numeric glob expansion is bounded to prevent excessive allocation.
- Does this need documentation?
- [x] No.
### Validation
- Connector module suites and packages: HMS 155 tests, Hive 491, Hudi 255, MaxCompute 155 (1 skipped), Iceberg 1,527 (5 skipped), and Paimon 726 (1 skipped); zero failures or errors in the final runs.
- Filesystem compatibility suites: 153 tests passed, including real Hadoop local-filesystem glob expansion and captured OSS requests.
- BE ASAN: all 29 `FileScannerV2Test` tests passed. Without the production fix, the new timestamp-partition test fails and the other 28 pass.
- Regression tests cover isolated expression-proxy loading, offline real DLF construction, OSS SPI discovery, canonical HMS storage-format metadata, schema deletion races, escaped Hudi paths, typed timestamp residuals, and explicit-offset precedence.
- Actual plugin ZIPs with a host Hive 3 QL jar: both offline DLF constructors, OSS SPI discovery, adapter/interface ownership, canonical format names, shared Jindo classes, and native initialization through both connector loaders passed.
- FE Checkstyle, clang-format 16, changed-line clang-tidy, build hygiene, shell syntax, and diff whitespace checks passed.
The remote external regression suites require external services and were not rerun locally. Standalone Hive Kryo serialization against the host QL jar still encounters a host `HiveConf` dependency gap; caller and vendor-bytecode checks confirm the affected name-based metadata paths do not invoke it. This PR does not claim to enable that separate path.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6182627341
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6140433361
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6146708671
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_77483a190e1adec4a2b0a53339d2cceeddbe262b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_77483a190e1adec4a2b0a53339d2cceeddbe262b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4257554103
##########
fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java:
##########
@@ -277,7 +279,7 @@ public List<FileEntry> listFiles(Location location) throws IOException {
}
private static boolean containsGlob(String path) {
- return path.contains("*") || path.contains("?") || path.contains("[");
+ return path.contains("*") || path.contains("?") || path.contains("[") || path.contains("{");
Review Comment:
Fixed in b3039f6f488. The cleanup consumer now enumerates the literal target using FileSystem.list, filters out directories, closes the iterator, and then removes the collected files. It retains the staging-directory exclusion. I audited the other callers before choosing this approach: backup and Parquet metadata still rely on listFiles glob behavior, so removing that behavior would introduce a separate regression.
Added an actual HiveConnectorTransaction regression over DFSFileSystem/Hadoop local storage with relative staging and table directory names containing {}, *, ? and []. It verifies the old file is removed, the new file is committed, and a neighboring glob-matching directory remains untouched. The brace-directory test failed before the fix. All existing filesystem discovery tests remain passing.
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -794,7 +799,8 @@ private List<ConnectorPartitionInfo> collectPartitions(HudiTableHandle handle, b
// HoodieTableMetaClient built just to read a timeline zone - plus a fallback that
// would have published epoch 0 as a partition's last-modified time for any
// non-numeric pin, the day it became reachable.
- return buildPartitionInfos(hmsNames, partKeyNames, latestInstantMillis(handle));
+ return buildPartitionInfos(hmsPartitionPaths(handle, hmsNames),
Review Comment:
Fixed in b3039f6f488. Physical-path parsing now takes an explicit encoding flag from HoodieTableConfig.getUrlEncodePartitioning(). Missing/false preserves literal percent sequences; true unescapes once. The flag is propagated through listing, pruning, pinned-partition filtering, snapshot scans and COW/MOR incremental scans, so the displayed values and scan values use the same contract. HMS name decoding for partition-object identity is kept separate.
Added tests using real persisted hoodie.properties for missing, false and true settings. They distinguish raw ':' and '%3A' in both listPartitions/listPartitionNames and applyFilter; the default-setting test reproduced the colon collision before the fix. Additional tests cover metadata/pinned listings, positional and prefixed paths, percent/slash/plus values, and MOR native/JNI ranges. The final scoped run passed 906 tests with no failures, errors or skips; FE Checkstyle passed for all 23 reactor modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4255125714
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+ private static final int MAX_EXPANSIONS = 10000;
+ private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+ private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+ private static final Pattern BRACE_GROUP = Pattern.compile("\\{([^{}]*)\\}");
Review Comment:
[P2] Preserve escaped braces inside numeric alternatives
For the valid pattern `data_{1..2,\{special\}}.csv`, this regex cannot match the outer group because its content includes the escaped opening brace. It matches only `\{special\}`, which the escape check then skips, leaving `1..2` unexpanded. `S3CompatibleFileSystem.compileGlobPattern` and `globListWithLimit` consequently match literal `data_1..2.csv` instead of `data_1.csv` and `data_2.csv`; the previous normalizer expanded this pattern correctly. Parse brace boundaries with escape awareness so literal braces in one alternative do not suppress ranges in another, and add this mixed escaped-brace case to the coverage.
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +927,42 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
return atPin;
}
+ private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+ List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+ .collect(Collectors.toList());
+ if (encodedNames.isEmpty()) {
+ return names;
+ }
+ // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+ // The storage location identifies the physical path regardless of the sync extractor's decoding
+ // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+ List<HmsPartitionInfo> partitions = hmsClient.getPartitions(
Review Comment:
[P2] Tolerate partitions deleted after the cached name listing
With `use_hive_sync_partition=true` and `meta.cache.hive.partition.enable=false` (the name cache remains enabled), cache a listing containing an encoded partition, then drop that partition in HMS. The next filtered query for a surviving partition still passes the stale encoded name here, and strict `getPartitions` throws for the missing result before `prunePartitionPaths` can discard it. Object-cache expiry or eviction causes the same failure, so retries can keep failing until the names cache is refreshed. Use an omission-tolerant lookup such as `getExistingPartitions` and associate surviving results by partition identity, omitting deleted names; retain errors for malformed results. Add a cached-name/deleted-partition case to the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141741467
<details>
<summary>TPC-H: <b>Total hot run time: 29362 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2cfee92005dbc607c8ac31897868be045670ef6f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17630 5684 5593 5593
q2 2152 335 249 249
q3 10213 1429 853 853
q4 4678 481 352 352
q5 7451 788 543 543
q6 189 191 166 166
q7 774 799 630 630
q8 9267 1345 1085 1085
q9 5726 4491 4465 4465
q10 6833 1295 1039 1039
q11 429 254 241 241
q12 634 414 290 290
q13 18077 3106 2409 2409
q14 281 273 254 254
q15 q16 756 733 688 688
q17 1207 753 584 584
q18 7135 6265 6227 6227
q19 1265 964 590 590
q20 421 344 240 240
q21 6288 2565 2673 2565
q22 416 351 299 299
Total cold run time: 101822 ms
Total hot run time: 29362 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6758 6709 6803 6709
q2 742 616 594 594
q3 5662 5737 5311 5311
q4 2037 2186 1574 1574
q5 5623 5566 5781 5566
q6 249 191 149 149
q7 2292 2026 1828 1828
q8 3143 2829 2777 2777
q9 8316 8167 8204 8167
q10 3673 3627 3264 3264
q11 605 408 374 374
q12 661 736 506 506
q13 2757 3091 2389 2389
q14 286 280 270 270
q15 q16 681 688 605 605
q17 7844 7014 6896 6896
q18 13156 12317 12979 12317
q19 919 816 836 816
q20 2189 2200 1946 1946
q21 5864 4705 4833 4705
q22 491 453 427 427
Total cold run time: 73948 ms
Total hot run time: 67190 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6150705719
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6184991198
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6148724144
<details>
<summary>TPC-H: <b>Total hot run time: 28899 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3039f6f48862c5ba02ef6ab5bb486daefb9c802, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17598 5618 5532 5532
q2 2097 339 253 253
q3 10227 1378 827 827
q4 4676 470 340 340
q5 7432 785 553 553
q6 187 184 151 151
q7 753 788 605 605
q8 9238 1302 1068 1068
q9 5706 4475 4428 4428
q10 6827 1281 1015 1015
q11 429 250 231 231
q12 634 395 285 285
q13 18051 3053 2377 2377
q14 279 266 241 241
q15 q16 736 722 683 683
q17 1170 728 567 567
q18 7070 6223 6168 6168
q19 1109 953 588 588
q20 384 328 223 223
q21 5534 2469 2560 2469
q22 403 337 295 295
Total cold run time: 100540 ms
Total hot run time: 28899 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6455 6550 6467 6467
q2 688 584 582 582
q3 5561 5818 5188 5188
q4 2355 2131 1579 1579
q5 5758 5738 5527 5527
q6 245 194 148 148
q7 2354 2009 1808 1808
q8 3118 2770 2738 2738
q9 8332 8014 7746 7746
q10 3654 3699 3223 3223
q11 596 395 378 378
q12 659 685 499 499
q13 2741 3057 2391 2391
q14 281 296 268 268
q15 q16 666 678 609 609
q17 7769 6911 6877 6877
q18 13144 12265 13025 12265
q19 939 787 788 787
q20 2177 2174 1937 1937
q21 5798 4692 4771 4692
q22 498 452 407 407
Total cold run time: 73788 ms
Total hot run time: 66116 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4255623057
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +927,42 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
return atPin;
}
+ private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+ List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+ .collect(Collectors.toList());
+ if (encodedNames.isEmpty()) {
+ return names;
+ }
+ // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+ // The storage location identifies the physical path regardless of the sync extractor's decoding
+ // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+ List<HmsPartitionInfo> partitions = hmsClient.getPartitions(
Review Comment:
[P2] Independently confirmed at `0d0163fc17d6151c60c9cc4a711b311034bb5a93`. The new location lookup runs before partition pruning, while `CachingHmsClient` caches names and partition objects independently. A stale encoded name therefore makes a query for a surviving partition fail when the object lookup misses; the merge-base path did not perform this strict lookup. Please use `getExistingPartitions`, associate returned objects by partition identity rather than list index, and omit confirmed deleted names while retaining malformed-result errors. Add a test with a cached encoded name whose HMS partition has been deleted.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+ private static final int MAX_EXPANSIONS = 10000;
+ private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+ private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+ private static final Pattern BRACE_GROUP = Pattern.compile("\\{([^{}]*)\\}");
Review Comment:
[P2] Independently confirmed at `0d0163fc17d6151c60c9cc4a711b311034bb5a93` against the merge-base normalizer. For `data_{1..2,\{special\}}.csv`, the outer group is not matched, so the numeric alternative remains literal and `data_1.csv` / `data_2.csv` are omitted. If the literal-brace alternative exists, discovery can still succeed with an incomplete file set. Please parse brace boundaries with escape awareness, reusing or extracting the existing `findClosingBrace` / `splitBraceAlternatives` logic where appropriate, and add this mixed numeric-range/literal-brace case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6143622861
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #68998:
URL: https://github.com/apache/doris/pull/68998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6138848620
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6138853541
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4257799798
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath, String excludedChildPat
}
deleteDir(dir);
}
- List<FileEntry> files = getFileSystem().listFiles(Location.of(targetPath));
+ // Storage implementations may treat listFiles arguments as globs. Overwrite must enumerate
+ // the literal target, or a table name such as table_{prod} can leave old rows behind.
+ List<FileEntry> files = new ArrayList<>();
+ try (FileIterator iterator = getFileSystem().list(Location.of(targetPath))) {
+ while (iterator.hasNext()) {
+ FileEntry file = iterator.next();
+ if (!file.isDirectory()) {
Review Comment:
[P1] Keep staged files out of recursive overwrite cleanup
For an unpartitioned Hive table at `abfss://workspace@onelake.dfs.fabric.microsoft.com/lake.lakehouse/Files/table_*` with `hive.staging_dir=.doris_staging`, `LocationPath` selects `FILE_HDFS`, so new files are staged beneath the target. `AzureFileSystem.list()` recursively emits those staged blobs as non-directories. The exclusion in the directory loop never protects them: this loop collects and deletes both staged input and old data before `prepareAlterTable` schedules the move, leaving the copy without its source and no restoration task for old rows. Previously the wildcard `listFiles()` branch rejected descendant keys, leaving the staging source intact even for an empty table. Exclude every file beneath `excludedChildPath`, or use literal nonrecursive enumeration, and cover this OneLake branch. This is distinct from the fixed DFS glob-matching issue in thread 4256817233. The literal-directory applicability follows [ADLS directory naming](https://learn.microsoft.com/en-us/rest/a
pi/storageservices/naming-and-referencing-containers--blobs--and-metadata#directory-names) and the documented [OneLake API parity](https://learn.microsoft.com/en-us/fabric/onelake/onelake-api-parity); the endpoint behavior is inferred from that parity and the traced code, not a live service test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6151245566
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256045503
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +928,52 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
return atPin;
}
+ private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+ List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+ .collect(Collectors.toList());
+ if (encodedNames.isEmpty()) {
+ return names;
+ }
+ // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+ // The storage location identifies the physical path regardless of the sync extractor's decoding
+ // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+ // Name and object caches expire independently. A deleted encoded partition must not fail
+ // a query before pruning; surviving objects must be associated by values, not list position.
+ List<HmsPartitionInfo> partitions = hmsClient.getExistingPartitions(
+ handle.getDbName(), handle.getTableName(), encodedNames);
+ Map<List<String>, String> namesByValues = new HashMap<>();
+ for (String name : encodedNames) {
+ List<String> values = Arrays.stream(name.split("/"))
+ .map(part -> HudiScanPlanProvider.unescapePathName(part.substring(part.indexOf('=') + 1)))
+ .collect(Collectors.toList());
+ namesByValues.put(values, name);
+ }
+ Map<String, String> paths = new HashMap<>();
+ for (HmsPartitionInfo partition : partitions) {
+ String name = namesByValues.get(partition.getValues());
+ paths.put(name, relativePartitionPath(handle.getBasePath(), partition.getLocation()));
+ }
+ return names.stream().filter(name -> name.indexOf('%') < 0 || paths.containsKey(name))
+ .map(name -> paths.getOrDefault(name, name)).collect(Collectors.toList());
+ }
+
+ private static String relativePartitionPath(String basePath, String location) {
+ if (location == null || location.isEmpty()) {
+ throw new DorisConnectorException("Hudi HMS partition has no storage location");
+ }
+ // Path quotes literal percent signs when creating its URI; getPath retains the Hudi path's
+ // own escapes. URI.create(location).getPath() would incorrectly consume that escaping layer.
+ URI base = new Path(basePath).toUri();
+ URI partition = new Path(location).toUri();
+ String prefix = base.getPath().endsWith("/") ? base.getPath() : base.getPath() + "/";
+ if (!Objects.equals(base.getScheme(), partition.getScheme())
Review Comment:
[P2] Accept equivalent storage schemes in Hudi partition locations
Normal Hive sync can produce a table at `s3a://b/t` and live partitions at `s3://b/t/...`: [Hudi 1.0.2 keeps an existing table when only the scheme changes](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java#L224-L235), while [new partition locations use the current sync base path](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java#L179-L193). Doris also maps both schemes to `S3AFileSystem`. With `use_hive_sync_partition=true`, an encoded partition such as `dt=a%253Ab` whose location is `s3://b/t/dt=a%3Ab` now throws "outside the table base path" against `s3a://b/t`. Because location resolution precedes pruning, even `WHERE dt='ok'` on an existing plain partition fails; the previous path pruned that HMS name without this check. Compare supported equivalent scheme identities while retaining the bucket and path containment ch
ecks, and cover mixed-scheme live partitions in both listing and pruning tests.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+ private static final int MAX_EXPANSIONS = 10000;
+ private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+ private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+
+ private GlobPattern() {
+ }
+
+ /**
+ * Expands {@code {N..M}} numeric range syntax in a glob pattern to the equivalent
+ * comma-separated alternation {@code {N,N+1,...,M}} understood by Hadoop and object-storage globs.
+ * Supports simple non-negative ranges like {@code {1..3}}, reverse ranges like {@code {3..1}},
+ * and mixed comma-separated patterns like {@code {1..2,3,1..3}} or {@code {Refrain,1..3}}.
+ * For simple (no-comma) brace groups, only non-negative ranges are expanded;
+ * negative-start ranges like {@code {-1..1}} are left unchanged.
+ * Duplicate values are removed. For example:
+ * <ul>
+ * <li>{@code data_{1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+ * <li>{@code data_{1..2,3,1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+ * <li>{@code data_{Refrain,1..3}.csv} → {@code data_{Refrain,1,2,3}.csv}</li>
+ * <li>{@code data_{-1..1}.csv} → unchanged (no expansion)</li>
+ * </ul>
+ */
+ public static String expandNumericRanges(String pattern) {
+ StringBuilder result = new StringBuilder();
+ Deque<BraceGroup> groups = new ArrayDeque<>();
+ ExpansionBudget budget = new ExpansionBudget();
+ boolean characterClass = false;
+ for (int i = 0; i < pattern.length(); i++) {
+ char c = pattern.charAt(i);
+ StringBuilder output = groups.isEmpty() ? result : groups.peek().segment;
+ // Escaped braces and commas are literal characters, even inside an alternative.
+ // Character classes likewise must not change the surrounding brace structure.
+ if (c == '\\') {
+ output.append(c);
+ if (i + 1 < pattern.length()) {
+ output.append(pattern.charAt(++i));
+ }
+ } else if (characterClass) {
+ output.append(c);
+ if (c == ']') {
+ characterClass = false;
+ }
+ } else if (c == '[') {
+ characterClass = true;
+ output.append(c);
+ } else if (c == '{') {
+ groups.push(new BraceGroup());
+ } else if (c == ',' && !groups.isEmpty()) {
+ groups.peek().finishSegment();
+ } else if (c == '}' && !groups.isEmpty()) {
+ BraceGroup group = groups.pop();
+ group.finishSegment();
+ String expanded = group.expand(budget);
+ (groups.isEmpty() ? result : groups.peek().segment).append('{').append(expanded).append('}');
+ } else {
+ output.append(c);
+ }
+ }
+ while (!groups.isEmpty()) {
+ BraceGroup group = groups.pop();
+ group.finishSegment();
+ (groups.isEmpty() ? result : groups.peek().segment)
+ .append('{').append(String.join(",", group.segments));
+ }
+ return result.toString();
+ }
+
+ private static final class ExpansionBudget {
+ private long combinations = 1;
+ private long expandedValues;
+ }
+
+ private static final class BraceGroup {
+ private final List<String> segments = new ArrayList<>();
+ private final StringBuilder segment = new StringBuilder();
+
+ private void finishSegment() {
+ segments.add(segment.toString());
+ segment.setLength(0);
+ }
+
+ private String expand(ExpansionBudget budget) {
+ if (segments.size() == 1 && !SIMPLE_RANGE.matcher(segments.get(0)).matches()) {
+ return segments.get(0);
+ }
+ Set<String> values = new LinkedHashSet<>();
+ boolean expanded = false;
+ for (String alternative : segments) {
+ Matcher range = RANGE.matcher(alternative.trim());
+ if (range.matches()) {
+ // Count before materializing, and use long arithmetic so an int endpoint
+ // cannot wrap the loop or turn a huge requested range into a small count.
+ long from = Integer.parseInt(range.group(1));
+ long to = Integer.parseInt(range.group(2));
+ budget.expandedValues += Math.abs(to - from) + 1;
+ checkLimit(budget.expandedValues);
+ long step = from <= to ? 1 : -1;
+ for (long value = from; step > 0 ? value <= to : value >= to; value += step) {
+ values.add(Long.toString(value));
+ }
+ expanded = true;
+ } else {
+ values.add(alternative.trim());
+ }
+ }
+ if (expanded) {
+ budget.combinations *= values.size();
Review Comment:
[P2] Count mutually exclusive ranges as alternatives
For `hdfs://namenode:8020/data_{a{1..101},b{1..101}}.csv`, the two inner ranges form 202 alternatives, but this shared product becomes `101 * 101 = 10201` and throws `IllegalArgumentException` even though only 202 numeric values are generated. Both `DFSFileSystem.listFiles` and `globListWithLimit` call this normalizer before `globStatus`, so discovery fails for a small valid nested glob. [Hadoop 3.4.2 supports nested brace alternatives](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GlobPattern.java#L102-L114). Track cardinality per brace expression, summing alternative branches and multiplying only independent concatenated groups, while retaining the allocation limit. Add a nested-alternative case alongside the existing Cartesian-product limit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6148922408
<details>
<summary>TPC-DS: <b>Total hot run time: 151904 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3039f6f48862c5ba02ef6ab5bb486daefb9c802, data reload: false
query5 4328 621 458 458
query6 412 200 196 196
query7 4804 446 228 228
query8 310 182 165 165
query9 8798 3954 3931 3931
query10 451 310 251 251
query11 5843 3436 3185 3185
query12 146 92 86 86
query13 1267 458 329 329
query14 6560 4811 4477 4477
query14_1 4280 4262 4169 4169
query15 210 204 186 186
query16 925 449 351 351
query17 854 672 559 559
query18 2417 425 310 310
query19 199 166 135 135
query20 81 83 78 78
query21 207 133 112 112
query22 13032 13040 12866 12866
query23 13155 12564 12125 12125
query23_1 12231 12352 12292 12292
query24 7131 820 456 456
query24_1 474 470 473 470
query25 536 423 370 370
query26 1268 262 141 141
query27 2759 445 275 275
query28 4574 1952 1956 1952
query29 1540 593 464 464
query30 295 217 183 183
query31 902 763 645 645
query32 144 96 91 91
query33 503 306 249 249
query34 882 846 514 514
query35 759 758 678 678
query36 796 791 740 740
query37 128 105 90 90
query38 1831 1733 1718 1718
query39 721 730 667 667
query39_1 673 663 683 663
query40 226 118 105 105
query41 74 70 72 70
query42 95 90 83 83
query43 369 375 326 326
query44 1314 707 704 704
query45 185 178 167 167
query46 852 955 572 572
query47 3055 2979 2864 2864
query48 291 317 221 221
query49 569 392 306 306
query50 672 268 225 225
query51 10708 10625 10678 10625
query52 86 82 71 71
query53 187 203 153 153
query54 248 182 173 173
query55 74 67 67 67
query56 222 209 194 194
query57 1613 1578 1580 1578
query58 281 260 255 255
query59 2343 2373 2129 2129
query60 269 238 213 213
query61 145 147 135 135
query62 397 345 282 282
query63 194 166 160 160
query64 2717 946 769 769
query65 3423 3384 3385 3384
query66 1774 404 295 295
query67 19890 20013 19851 19851
query68 2979 993 626 626
query69 382 283 250 250
query70 939 839 807 807
query71 290 226 206 206
query72 2697 2500 2223 2223
query73 516 518 297 297
query74 4607 4471 4267 4267
query75 2327 2439 1930 1930
query76 2307 1014 606 606
query77 356 393 301 301
query78 9016 8977 8499 8499
query79 977 895 515 515
query80 533 431 334 334
query81 527 320 275 275
query82 577 136 114 114
query83 302 215 174 174
query84 306 119 99 99
query85 790 423 357 357
query86 317 235 230 230
query87 1994 1929 1802 1802
query88 3609 2710 2685 2685
query89 344 296 260 260
query90 1903 183 184 183
query91 167 149 126 126
query92 92 87 88 87
query93 1018 995 578 578
query94 446 303 271 271
query95 583 405 320 320
query96 661 553 231 231
query97 2468 2433 2331 2331
query98 154 147 143 143
query99 802 741 643 643
Total cold run time: 233487 ms
Total hot run time: 151904 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6140435893
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6143622150
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6183261390
<details>
<summary>ClickBench: <b>Total hot run time: 24.63 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d715bf7dedee46773863d4ef767b1cde328a913d, data reload: false
query1 0.00 0.00 0.01
query2 0.10 0.05 0.04
query3 0.26 0.12 0.11
query4 1.61 0.13 0.12
query5 0.25 0.25 0.24
query6 1.17 0.69 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.35 0.35
query10 0.57 0.56 0.58
query11 0.20 0.14 0.13
query12 0.18 0.14 0.14
query13 0.50 0.51 0.49
query14 1.04 1.01 1.01
query15 0.63 0.61 0.61
query16 0.34 0.34 0.35
query17 1.14 1.20 1.14
query18 0.23 0.21 0.20
query19 2.08 1.97 2.01
query20 0.02 0.01 0.01
query21 15.49 0.26 0.14
query22 4.45 0.05 0.05
query23 16.25 0.32 0.13
query24 2.93 0.57 0.36
query25 0.13 0.05 0.05
query26 1.04 0.24 0.16
query27 0.05 0.03 0.04
query28 1.92 0.64 0.41
query29 12.50 4.20 3.35
query30 0.28 0.15 0.13
query31 2.78 0.57 0.34
query32 3.23 0.62 0.52
query33 3.16 3.26 3.33
query34 15.55 4.00 3.41
query35 3.40 3.34 3.32
query36 0.57 0.43 0.44
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.18 0.15 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 95.06 s
Total hot run time: 24.63 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6139588509
<details>
<summary>ClickBench: <b>Total hot run time: 24.41 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d0163fc17d6151c60c9cc4a711b311034bb5a93, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.04
query3 0.27 0.12 0.10
query4 1.61 0.12 0.12
query5 0.27 0.24 0.24
query6 1.20 0.68 0.70
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.42 0.35 0.35
query10 0.56 0.56 0.56
query11 0.20 0.15 0.14
query12 0.18 0.14 0.14
query13 0.48 0.48 0.48
query14 0.98 0.96 0.97
query15 0.64 0.61 0.60
query16 0.35 0.35 0.34
query17 1.13 1.11 1.15
query18 0.23 0.21 0.21
query19 2.11 1.97 2.01
query20 0.02 0.01 0.02
query21 15.49 0.28 0.14
query22 4.50 0.05 0.05
query23 16.27 0.33 0.12
query24 3.19 0.56 0.42
query25 0.13 0.07 0.04
query26 1.33 0.25 0.15
query27 0.04 0.05 0.04
query28 2.04 0.62 0.40
query29 12.51 4.22 3.28
query30 0.29 0.15 0.15
query31 2.77 0.59 0.35
query32 3.23 0.62 0.53
query33 3.16 3.17 3.31
query34 15.58 4.01 3.38
query35 3.38 3.31 3.36
query36 0.56 0.43 0.42
query37 0.10 0.07 0.06
query38 0.05 0.05 0.04
query39 0.03 0.02 0.03
query40 0.17 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.82 s
Total hot run time: 24.41 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141992016
<details>
<summary>ClickBench: <b>Total hot run time: 24.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2cfee92005dbc607c8ac31897868be045670ef6f, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.04
query3 0.27 0.12 0.12
query4 1.62 0.13 0.12
query5 0.28 0.25 0.26
query6 1.16 0.70 0.69
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.41 0.36 0.35
query10 0.60 0.61 0.57
query11 0.20 0.14 0.13
query12 0.18 0.15 0.14
query13 0.50 0.50 0.50
query14 0.98 0.97 0.95
query15 0.65 0.62 0.59
query16 0.35 0.34 0.34
query17 1.14 1.10 1.14
query18 0.22 0.20 0.21
query19 2.12 2.03 1.96
query20 0.01 0.01 0.01
query21 15.50 0.27 0.15
query22 4.41 0.06 0.05
query23 16.24 0.34 0.12
query24 3.11 0.56 0.39
query25 0.12 0.05 0.06
query26 1.03 0.25 0.14
query27 0.05 0.05 0.04
query28 1.92 0.62 0.39
query29 12.57 4.23 3.33
query30 0.28 0.14 0.13
query31 2.77 0.58 0.34
query32 3.23 0.62 0.51
query33 3.22 3.25 3.30
query34 15.42 3.96 3.39
query35 3.32 3.36 3.36
query36 0.56 0.47 0.49
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.02
query40 0.17 0.15 0.14
query41 0.09 0.03 0.02
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 95.17 s
Total hot run time: 24.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141852946
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6144503486
<details>
<summary>TPC-H: <b>Total hot run time: 28793 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77483a190e1adec4a2b0a53339d2cceeddbe262b, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17571 5658 5530 5530
q2 2102 302 242 242
q3 10242 1386 826 826
q4 4681 475 366 366
q5 7425 794 537 537
q6 188 199 151 151
q7 780 811 612 612
q8 9252 1338 1078 1078
q9 5690 4463 4411 4411
q10 6816 1309 1025 1025
q11 423 254 251 251
q12 634 409 288 288
q13 18059 3098 2379 2379
q14 279 270 245 245
q15 q16 742 732 663 663
q17 1202 745 570 570
q18 7029 6215 6167 6167
q19 1122 953 626 626
q20 375 338 227 227
q21 5525 2329 2358 2329
q22 360 311 270 270
Total cold run time: 100497 ms
Total hot run time: 28793 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6121 6132 6104 6104
q2 670 587 564 564
q3 5222 5369 4931 4931
q4 2017 2131 1570 1570
q5 5250 5187 5893 5187
q6 250 205 157 157
q7 2291 2082 1898 1898
q8 3186 2781 2775 2775
q9 8246 8197 8234 8197
q10 3965 3785 3359 3359
q11 679 465 391 391
q12 690 743 548 548
q13 2978 3223 2540 2540
q14 340 335 294 294
q15 q16 704 755 668 668
q17 8303 7617 7469 7469
q18 13200 12274 13079 12274
q19 920 808 795 795
q20 2187 2174 1951 1951
q21 5781 4737 4818 4737
q22 503 473 393 393
Total cold run time: 73503 ms
Total hot run time: 66802 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6144665063
<details>
<summary>TPC-DS: <b>Total hot run time: 152237 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77483a190e1adec4a2b0a53339d2cceeddbe262b, data reload: false
query5 4298 584 446 446
query6 426 231 186 186
query7 4801 442 224 224
query8 339 179 166 166
query9 8756 4023 4057 4023
query10 494 319 254 254
query11 6037 3424 3186 3186
query12 144 91 85 85
query13 1259 438 322 322
query14 6551 4863 4494 4494
query14_1 4290 4309 4249 4249
query15 206 203 176 176
query16 939 459 426 426
query17 876 659 552 552
query18 2425 433 324 324
query19 189 170 160 160
query20 81 80 79 79
query21 206 140 115 115
query22 13027 13198 12896 12896
query23 13189 12533 12239 12239
query23_1 12169 12155 12261 12155
query24 7090 828 469 469
query24_1 475 476 480 476
query25 532 427 377 377
query26 1257 263 139 139
query27 2763 444 277 277
query28 4588 1955 1987 1955
query29 1555 584 439 439
query30 289 211 187 187
query31 872 760 643 643
query32 147 103 93 93
query33 516 295 236 236
query34 930 850 488 488
query35 748 766 644 644
query36 841 832 752 752
query37 135 102 92 92
query38 1797 1747 1685 1685
query39 712 721 697 697
query39_1 664 669 700 669
query40 213 115 92 92
query41 67 66 65 65
query42 86 85 81 81
query43 362 381 328 328
query44 1325 701 687 687
query45 181 173 168 168
query46 850 1025 580 580
query47 3013 2954 2885 2885
query48 296 312 216 216
query49 570 402 281 281
query50 676 283 212 212
query51 10744 10350 10379 10350
query52 76 83 71 71
query53 186 208 149 149
query54 236 189 173 173
query55 74 77 63 63
query56 231 231 206 206
query57 1618 1615 1576 1576
query58 284 288 249 249
query59 2365 2401 2124 2124
query60 283 238 222 222
query61 147 151 142 142
query62 395 364 295 295
query63 204 163 157 157
query64 2731 936 781 781
query65 3437 3372 3367 3367
query66 1773 417 307 307
query67 20074 20109 19949 19949
query68 2897 991 637 637
query69 383 291 251 251
query70 955 879 817 817
query71 295 233 218 218
query72 2740 2570 2270 2270
query73 515 546 299 299
query74 4574 4471 4330 4330
query75 2335 2294 1972 1972
query76 2289 1034 597 597
query77 355 407 310 310
query78 8915 8998 8492 8492
query79 858 889 528 528
query80 479 434 365 365
query81 526 327 271 271
query82 209 135 115 115
query83 195 196 179 179
query84 292 112 105 105
query85 779 431 392 392
query86 255 242 226 226
query87 1975 1951 1819 1819
query88 3601 2724 2672 2672
query89 328 297 257 257
query90 2098 180 181 180
query91 158 154 122 122
query92 99 84 93 84
query93 992 1008 572 572
query94 453 285 287 285
query95 582 401 323 323
query96 667 521 229 229
query97 2443 2395 2361 2361
query98 161 158 150 150
query99 761 775 652 652
Total cold run time: 232896 ms
Total hot run time: 152237 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6150510724
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6151948565
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/14)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6139521013
<details>
<summary>TPC-DS: <b>Total hot run time: 151418 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d0163fc17d6151c60c9cc4a711b311034bb5a93, data reload: false
query5 4330 580 449 449
query6 422 213 203 203
query7 4802 440 227 227
query8 312 181 170 170
query9 8743 3994 3994 3994
query10 453 311 251 251
query11 5885 3451 3199 3199
query12 157 88 85 85
query13 1283 450 326 326
query14 6530 4784 4460 4460
query14_1 4229 4258 4225 4225
query15 208 211 182 182
query16 935 445 423 423
query17 874 645 539 539
query18 2416 422 308 308
query19 201 163 135 135
query20 81 82 82 82
query21 210 137 113 113
query22 12995 13023 12786 12786
query23 13253 12618 12080 12080
query23_1 12269 12101 12211 12101
query24 7115 858 496 496
query24_1 466 456 467 456
query25 508 399 332 332
query26 1258 252 134 134
query27 2771 435 268 268
query28 4628 1965 1953 1953
query29 1553 574 424 424
query30 288 213 183 183
query31 888 755 655 655
query32 141 116 89 89
query33 518 293 242 242
query34 921 845 491 491
query35 735 743 651 651
query36 829 818 726 726
query37 125 97 91 91
query38 1808 1740 1682 1682
query39 706 719 683 683
query39_1 675 681 674 674
query40 212 119 96 96
query41 67 63 62 62
query42 92 89 79 79
query43 367 371 318 318
query44 1321 697 698 697
query45 178 192 165 165
query46 836 973 563 563
query47 2966 2944 2939 2939
query48 299 302 221 221
query49 571 394 321 321
query50 667 267 208 208
query51 10319 10532 10311 10311
query52 82 81 74 74
query53 181 208 150 150
query54 245 189 182 182
query55 75 66 63 63
query56 240 207 204 204
query57 1636 1598 1545 1545
query58 287 255 257 255
query59 2375 2399 2125 2125
query60 276 233 214 214
query61 146 148 140 140
query62 388 339 284 284
query63 193 161 178 161
query64 2735 908 773 773
query65 3436 3349 3378 3349
query66 1779 428 321 321
query67 20301 20298 19889 19889
query68 3190 987 628 628
query69 379 303 251 251
query70 935 854 816 816
query71 294 228 211 211
query72 2786 2612 2309 2309
query73 520 533 292 292
query74 4602 4445 4266 4266
query75 2335 2293 1929 1929
query76 2375 1003 620 620
query77 353 393 301 301
query78 9117 8914 8395 8395
query79 877 837 516 516
query80 473 409 347 347
query81 526 327 277 277
query82 213 135 105 105
query83 190 197 180 180
query84 290 120 108 108
query85 759 431 366 366
query86 257 239 235 235
query87 2000 1961 1842 1842
query88 3604 2726 2653 2653
query89 326 298 257 257
query90 2101 189 182 182
query91 153 146 125 125
query92 99 87 85 85
query93 978 981 570 570
query94 434 302 293 293
query95 594 343 317 317
query96 674 519 232 232
query97 2461 2424 2332 2332
query98 160 151 152 151
query99 770 757 646 646
Total cold run time: 233411 ms
Total hot run time: 151418 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141853432
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6138850813
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6147701031
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6144746627
<details>
<summary>ClickBench: <b>Total hot run time: 24.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77483a190e1adec4a2b0a53339d2cceeddbe262b, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.13 0.13
query5 0.26 0.25 0.23
query6 1.18 0.69 0.67
query7 0.04 0.01 0.01
query8 0.05 0.03 0.03
query9 0.42 0.36 0.37
query10 0.58 0.58 0.59
query11 0.22 0.15 0.14
query12 0.18 0.14 0.15
query13 0.51 0.50 0.51
query14 0.98 0.96 0.97
query15 0.65 0.62 0.61
query16 0.32 0.32 0.36
query17 1.16 1.16 1.15
query18 0.23 0.21 0.20
query19 2.04 1.97 2.04
query20 0.02 0.01 0.01
query21 15.49 0.27 0.14
query22 4.45 0.05 0.05
query23 16.27 0.33 0.12
query24 3.10 0.65 0.43
query25 0.12 0.04 0.04
query26 1.64 0.24 0.14
query27 0.05 0.04 0.04
query28 2.56 0.61 0.42
query29 12.51 4.23 3.29
query30 0.28 0.13 0.12
query31 2.77 0.60 0.34
query32 3.25 0.63 0.53
query33 3.20 3.22 3.17
query34 15.43 4.07 3.38
query35 3.36 3.36 3.36
query36 0.56 0.44 0.43
query37 0.09 0.06 0.06
query38 0.04 0.04 0.03
query39 0.04 0.02 0.02
query40 0.18 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 96.38 s
Total hot run time: 24.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4278222805
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath, String excludedChildPat
}
deleteDir(dir);
}
- List<FileEntry> files = getFileSystem().listFiles(Location.of(targetPath));
+ // Storage implementations may treat listFiles arguments as globs. Overwrite must enumerate
+ // the literal target, or a table name such as table_{prod} can leave old rows behind.
+ List<FileEntry> files = new ArrayList<>();
+ try (FileIterator iterator = getFileSystem().list(Location.of(targetPath))) {
+ while (iterator.hasNext()) {
+ FileEntry file = iterator.next();
+ if (!file.isDirectory()) {
Review Comment:
Fixed in d715bf7dede. Both cleanup loops now exclude the staging root and every descendant, using the existing filesystem-aware equality and slash-boundary containment helpers. The directory loop also excludes nested staging directories if a provider lists them directly. Literal target enumeration and existing glob discovery behavior remain unchanged.
Added a transaction regression using real AzureFileSystem enumeration and copy-then-delete rename with an in-memory paginated AzureObjStorage. Before the fix it fails with FileNotFoundException for the staged source, including an otherwise empty table. The test covers 12 combinations of literal target names, empty/nonempty old data and directory-listing shapes. It checks that new data is published, other-query staging and adjacent tables survive, and direct/nested old files plus .doris_staging_backup are cleaned up. No live OneLake endpoint was used.
Validation: 604 scoped Hive/Azure/HDFS/shared-HMS tests passed with zero failures, errors or skips; FE Checkstyle passed in 21 reactor modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6183205515
<details>
<summary>TPC-DS: <b>Total hot run time: 153402 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d715bf7dedee46773863d4ef767b1cde328a913d, data reload: false
query5 4306 593 474 474
query6 426 210 195 195
query7 4810 450 231 231
query8 319 178 172 172
query9 8779 4059 4071 4059
query10 499 320 269 269
query11 6076 3777 3553 3553
query12 147 91 86 86
query13 1249 445 326 326
query14 6100 4067 3787 3787
query14_1 3568 3588 3569 3569
query15 206 200 188 188
query16 954 472 431 431
query17 852 661 549 549
query18 2421 445 313 313
query19 213 159 137 137
query20 86 92 82 82
query21 208 143 115 115
query22 13000 12998 12972 12972
query23 13518 12810 12461 12461
query23_1 12670 12847 12797 12797
query24 7095 849 481 481
query24_1 494 486 481 481
query25 539 432 367 367
query26 1366 267 138 138
query27 2765 435 268 268
query28 4612 1939 1943 1939
query29 1564 573 438 438
query30 310 206 184 184
query31 1008 935 807 807
query32 137 90 83 83
query33 505 295 241 241
query34 893 852 504 504
query35 744 750 667 667
query36 817 790 717 717
query37 125 101 89 89
query38 1814 1721 1731 1721
query39 795 796 748 748
query39_1 740 728 745 728
query40 215 114 94 94
query41 65 62 66 62
query42 98 85 82 82
query43 369 392 334 334
query44 1320 696 687 687
query45 180 174 162 162
query46 853 981 567 567
query47 3007 3020 2795 2795
query48 316 345 219 219
query49 576 398 301 301
query50 710 288 214 214
query51 10483 10429 10535 10429
query52 79 96 74 74
query53 194 205 151 151
query54 228 189 174 174
query55 69 64 64 64
query56 232 213 218 213
query57 1614 1597 1565 1565
query58 286 265 258 258
query59 2083 2188 1955 1955
query60 283 241 224 224
query61 145 146 143 143
query62 399 340 284 284
query63 201 162 171 162
query64 2676 909 786 786
query65 3341 3255 3297 3255
query66 1772 411 302 302
query67 20336 20077 20104 20077
query68 2954 1043 620 620
query69 380 287 241 241
query70 847 809 780 780
query71 302 244 241 241
query72 2701 2759 2468 2468
query73 540 566 305 305
query74 4607 4483 4322 4322
query75 2331 2303 1963 1963
query76 2308 1037 630 630
query77 354 383 295 295
query78 9683 9701 9179 9179
query79 1028 911 528 528
query80 549 456 373 373
query81 522 326 276 276
query82 592 147 117 117
query83 318 209 242 209
query84 315 125 97 97
query85 802 416 373 373
query86 309 239 233 233
query87 2011 1962 1827 1827
query88 3664 2726 2710 2710
query89 340 296 261 261
query90 1909 192 184 184
query91 156 150 124 124
query92 99 90 91 90
query93 1064 1033 578 578
query94 453 319 275 275
query95 596 339 383 339
query96 675 547 234 234
query97 2455 2436 2341 2341
query98 161 153 165 153
query99 778 761 642 642
Total cold run time: 234492 ms
Total hot run time: 153402 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4278181403
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath, String excludedChildPat
}
deleteDir(dir);
}
- List<FileEntry> files = getFileSystem().listFiles(Location.of(targetPath));
+ // Storage implementations may treat listFiles arguments as globs. Overwrite must enumerate
+ // the literal target, or a table name such as table_{prod} can leave old rows behind.
+ List<FileEntry> files = new ArrayList<>();
+ try (FileIterator iterator = getFileSystem().list(Location.of(targetPath))) {
+ while (iterator.hasNext()) {
+ FileEntry file = iterator.next();
+ if (!file.isDirectory()) {
Review Comment:
[P1] Independently confirmed by tracing the code at `b3039f6f48862c5ba02ef6ab5bb486daefb9c802`. This does not require an upgrade or metadata from an older master build: a fresh deployment with a newly created unpartitioned Hive table at the OneLake path above and relative `hive.staging_dir=.doris_staging` can reach the same path.
`prepareAlterTable` calls this cleanup before `asyncRenameFiles`. Unlike the immediate-child DFS listing covered by the new test, `AzureFileSystem.list()` returns descendant blobs, including `.doris_staging/.../new.parquet`, with `isDirectory=false`. The directory-loop exclusion therefore does not protect the staged source; the file loop deletes both it and existing table files. The subsequent move fails, and this relative-staging branch has no old-data restoration task.
Please exclude the staging root and all its descendants from the file deletion set, using path-boundary-aware comparisons such as the existing `pathsEqual` / `isSubDirectory` helpers. Add a transaction regression using the real `AzureFileSystem` with a recording object store, verifying that staged inputs survive cleanup and the overwrite commits successfully. This conclusion is source-based; I have not run a live OneLake test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6183090405
<details>
<summary>TPC-H: <b>Total hot run time: 29201 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d715bf7dedee46773863d4ef767b1cde328a913d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17628 5653 5550 5550
q2 2084 319 271 271
q3 10253 1400 864 864
q4 4685 471 345 345
q5 7447 801 531 531
q6 186 186 158 158
q7 760 800 606 606
q8 9251 1380 1077 1077
q9 5536 4266 4252 4252
q10 6822 1309 1019 1019
q11 429 258 236 236
q12 631 409 284 284
q13 18087 3633 2948 2948
q14 284 268 247 247
q15 q16 739 732 670 670
q17 1168 751 572 572
q18 7080 6239 6188 6188
q19 1108 951 617 617
q20 375 332 228 228
q21 4974 2254 2382 2254
q22 360 312 284 284
Total cold run time: 99887 ms
Total hot run time: 29201 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6198 6267 6340 6267
q2 712 591 555 555
q3 5214 5408 4972 4972
q4 2040 2139 1590 1590
q5 5300 5456 5790 5456
q6 260 200 164 164
q7 2379 2024 1943 1943
q8 3112 2846 2803 2803
q9 8201 8050 8193 8050
q10 3814 3708 3324 3324
q11 664 469 404 404
q12 697 740 558 558
q13 3517 3939 3310 3310
q14 310 334 288 288
q15 q16 711 742 665 665
q17 8473 7543 6960 6960
q18 13217 12266 13092 12266
q19 952 829 803 803
q20 2191 2183 1935 1935
q21 5891 4792 4969 4792
q22 540 479 396 396
Total cold run time: 74393 ms
Total hot run time: 67501 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6184844864
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_d715bf7dedee46773863d4ef767b1cde328a913d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.07% (35621/46217) |
| Line Coverage | 62.40% (403698/646949) |
| Region Coverage | 58.83% (341102/579781) |
| Branch Coverage | 59.73% (157107/263010) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6182627946
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6146574981
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68998_b3039f6f48862c5ba02ef6ab5bb486daefb9c802_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6139398135
<details>
<summary>TPC-H: <b>Total hot run time: 29129 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d0163fc17d6151c60c9cc4a711b311034bb5a93, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17622 5624 5540 5540
q2 2076 294 237 237
q3 10288 1406 847 847
q4 4679 478 344 344
q5 7433 799 533 533
q6 187 187 153 153
q7 756 793 615 615
q8 9260 1330 1053 1053
q9 5647 4460 4452 4452
q10 6804 1308 1027 1027
q11 430 261 240 240
q12 633 408 285 285
q13 18048 3063 2383 2383
q14 282 273 250 250
q15 q16 748 731 680 680
q17 1304 752 579 579
q18 7189 6247 6214 6214
q19 1112 957 626 626
q20 390 336 231 231
q21 5857 2541 2576 2541
q22 397 356 299 299
Total cold run time: 101142 ms
Total hot run time: 29129 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6664 6639 6722 6639
q2 696 582 575 575
q3 5657 5707 5268 5268
q4 2035 2131 1586 1586
q5 5848 5694 5946 5694
q6 245 185 143 143
q7 2248 2056 1852 1852
q8 3089 2819 2678 2678
q9 8182 8149 7798 7798
q10 3664 3585 3285 3285
q11 586 399 365 365
q12 659 706 490 490
q13 2755 3052 2396 2396
q14 283 294 271 271
q15 q16 657 687 613 613
q17 7828 7246 6905 6905
q18 13096 12272 13095 12272
q19 904 799 799 799
q20 2199 2188 1939 1939
q21 5768 4702 4821 4702
q22 510 472 392 392
Total cold run time: 73573 ms
Total hot run time: 66662 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256354409
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +928,52 @@ private List<ConnectorPartitionInfo> restrictToPin(HudiTableHandle handle, List<
return atPin;
}
+ private List<String> hmsPartitionPaths(HudiTableHandle handle, List<String> names) {
+ List<String> encodedNames = names.stream().filter(name -> name.indexOf('%') >= 0)
+ .collect(Collectors.toList());
+ if (encodedNames.isEmpty()) {
+ return names;
+ }
+ // Hive sync can register an already escaped Hudi value, which HMS escapes again in its name.
+ // The storage location identifies the physical path regardless of the sync extractor's decoding
+ // policy. Decode that path only in parsePartitionValues, preserving literal percent sequences.
+ // Name and object caches expire independently. A deleted encoded partition must not fail
+ // a query before pruning; surviving objects must be associated by values, not list position.
+ List<HmsPartitionInfo> partitions = hmsClient.getExistingPartitions(
+ handle.getDbName(), handle.getTableName(), encodedNames);
+ Map<List<String>, String> namesByValues = new HashMap<>();
+ for (String name : encodedNames) {
+ List<String> values = Arrays.stream(name.split("/"))
+ .map(part -> HudiScanPlanProvider.unescapePathName(part.substring(part.indexOf('=') + 1)))
+ .collect(Collectors.toList());
+ namesByValues.put(values, name);
+ }
+ Map<String, String> paths = new HashMap<>();
+ for (HmsPartitionInfo partition : partitions) {
+ String name = namesByValues.get(partition.getValues());
+ paths.put(name, relativePartitionPath(handle.getBasePath(), partition.getLocation()));
+ }
+ return names.stream().filter(name -> name.indexOf('%') < 0 || paths.containsKey(name))
+ .map(name -> paths.getOrDefault(name, name)).collect(Collectors.toList());
+ }
+
+ private static String relativePartitionPath(String basePath, String location) {
+ if (location == null || location.isEmpty()) {
+ throw new DorisConnectorException("Hudi HMS partition has no storage location");
+ }
+ // Path quotes literal percent signs when creating its URI; getPath retains the Hudi path's
+ // own escapes. URI.create(location).getPath() would incorrectly consume that escaping layer.
+ URI base = new Path(basePath).toUri();
+ URI partition = new Path(location).toUri();
+ String prefix = base.getPath().endsWith("/") ? base.getPath() : base.getPath() + "/";
+ if (!Objects.equals(base.getScheme(), partition.getScheme())
Review Comment:
[P2] Independently confirmed at `2cfee92005dbc607c8ac31897868be045670ef6f`. `getTableHandle` takes the base path from HMS, and both partition listing and `applyFilter` resolve encoded partition locations before pruning. A table at `s3a://b/t` with a live partition at `s3://b/t/dt=a%3Ab` therefore fails this check, including queries targeting another plain partition. This can arise from normal Hive sync: table-location comparison ignores the scheme, while new partitions use the current sync base path. Doris also maps `fs.s3.impl` and `fs.s3a.impl` to `S3AFileSystem`. Please compare supported equivalent storage schemes here, retaining authority/bucket and directory containment checks, and add mixed-scheme listing and pruning tests in both directions.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob listing. */
+public final class GlobPattern {
+ private static final int MAX_EXPANSIONS = 10000;
+ private static final Pattern RANGE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+ private static final Pattern SIMPLE_RANGE = Pattern.compile("\\d+\\.\\.\\d+");
+
+ private GlobPattern() {
+ }
+
+ /**
+ * Expands {@code {N..M}} numeric range syntax in a glob pattern to the equivalent
+ * comma-separated alternation {@code {N,N+1,...,M}} understood by Hadoop and object-storage globs.
+ * Supports simple non-negative ranges like {@code {1..3}}, reverse ranges like {@code {3..1}},
+ * and mixed comma-separated patterns like {@code {1..2,3,1..3}} or {@code {Refrain,1..3}}.
+ * For simple (no-comma) brace groups, only non-negative ranges are expanded;
+ * negative-start ranges like {@code {-1..1}} are left unchanged.
+ * Duplicate values are removed. For example:
+ * <ul>
+ * <li>{@code data_{1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+ * <li>{@code data_{1..2,3,1..3}.csv} → {@code data_{1,2,3}.csv}</li>
+ * <li>{@code data_{Refrain,1..3}.csv} → {@code data_{Refrain,1,2,3}.csv}</li>
+ * <li>{@code data_{-1..1}.csv} → unchanged (no expansion)</li>
+ * </ul>
+ */
+ public static String expandNumericRanges(String pattern) {
+ StringBuilder result = new StringBuilder();
+ Deque<BraceGroup> groups = new ArrayDeque<>();
+ ExpansionBudget budget = new ExpansionBudget();
+ boolean characterClass = false;
+ for (int i = 0; i < pattern.length(); i++) {
+ char c = pattern.charAt(i);
+ StringBuilder output = groups.isEmpty() ? result : groups.peek().segment;
+ // Escaped braces and commas are literal characters, even inside an alternative.
+ // Character classes likewise must not change the surrounding brace structure.
+ if (c == '\\') {
+ output.append(c);
+ if (i + 1 < pattern.length()) {
+ output.append(pattern.charAt(++i));
+ }
+ } else if (characterClass) {
+ output.append(c);
+ if (c == ']') {
+ characterClass = false;
+ }
+ } else if (c == '[') {
+ characterClass = true;
+ output.append(c);
+ } else if (c == '{') {
+ groups.push(new BraceGroup());
+ } else if (c == ',' && !groups.isEmpty()) {
+ groups.peek().finishSegment();
+ } else if (c == '}' && !groups.isEmpty()) {
+ BraceGroup group = groups.pop();
+ group.finishSegment();
+ String expanded = group.expand(budget);
+ (groups.isEmpty() ? result : groups.peek().segment).append('{').append(expanded).append('}');
+ } else {
+ output.append(c);
+ }
+ }
+ while (!groups.isEmpty()) {
+ BraceGroup group = groups.pop();
+ group.finishSegment();
+ (groups.isEmpty() ? result : groups.peek().segment)
+ .append('{').append(String.join(",", group.segments));
+ }
+ return result.toString();
+ }
+
+ private static final class ExpansionBudget {
+ private long combinations = 1;
+ private long expandedValues;
+ }
+
+ private static final class BraceGroup {
+ private final List<String> segments = new ArrayList<>();
+ private final StringBuilder segment = new StringBuilder();
+
+ private void finishSegment() {
+ segments.add(segment.toString());
+ segment.setLength(0);
+ }
+
+ private String expand(ExpansionBudget budget) {
+ if (segments.size() == 1 && !SIMPLE_RANGE.matcher(segments.get(0)).matches()) {
+ return segments.get(0);
+ }
+ Set<String> values = new LinkedHashSet<>();
+ boolean expanded = false;
+ for (String alternative : segments) {
+ Matcher range = RANGE.matcher(alternative.trim());
+ if (range.matches()) {
+ // Count before materializing, and use long arithmetic so an int endpoint
+ // cannot wrap the loop or turn a huge requested range into a small count.
+ long from = Integer.parseInt(range.group(1));
+ long to = Integer.parseInt(range.group(2));
+ budget.expandedValues += Math.abs(to - from) + 1;
+ checkLimit(budget.expandedValues);
+ long step = from <= to ? 1 : -1;
+ for (long value = from; step > 0 ? value <= to : value >= to; value += step) {
+ values.add(Long.toString(value));
+ }
+ expanded = true;
+ } else {
+ values.add(alternative.trim());
+ }
+ }
+ if (expanded) {
+ budget.combinations *= values.size();
Review Comment:
[P2] Independently confirmed at `2cfee92005dbc607c8ac31897868be045670ef6f`. For `data_{a{1..101},b{1..101}}.csv`, the two inner groups share this budget: closing the first sets `combinations=101`, and closing the second sets it to `10201`, although the branches are mutually exclusive and produce only 202 candidates (`expandedValues` is also 202). Both DFS discovery entry points fail before Hadoop sees the pattern. Please track candidate counts per expression, summing alternative branches and multiplying independent concatenated groups, while retaining the global allocation limit. Add this 202-candidate success case and keep the `{1..101}/{1..101}` Cartesian-product rejection test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6141911686
<details>
<summary>TPC-DS: <b>Total hot run time: 152533 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2cfee92005dbc607c8ac31897868be045670ef6f, data reload: false
query5 4320 608 457 457
query6 426 222 183 183
query7 4805 449 220 220
query8 319 183 164 164
query9 8782 4013 3941 3941
query10 474 302 260 260
query11 5996 3420 3172 3172
query12 145 86 84 84
query13 1253 450 309 309
query14 6520 4796 4492 4492
query14_1 4308 4301 4244 4244
query15 199 196 180 180
query16 942 422 400 400
query17 845 628 531 531
query18 2402 411 288 288
query19 194 156 124 124
query20 78 79 76 76
query21 210 136 121 121
query22 12972 13070 12859 12859
query23 13252 12453 12271 12271
query23_1 12257 12332 12340 12332
query24 7072 800 451 451
query24_1 471 469 487 469
query25 507 395 334 334
query26 1254 255 134 134
query27 2775 439 268 268
query28 4629 1937 1941 1937
query29 1510 579 439 439
query30 330 218 186 186
query31 877 771 645 645
query32 152 108 82 82
query33 505 297 230 230
query34 918 847 487 487
query35 735 746 641 641
query36 834 840 739 739
query37 138 101 90 90
query38 1781 1764 1678 1678
query39 721 707 658 658
query39_1 669 693 656 656
query40 218 113 92 92
query41 64 63 65 63
query42 84 85 82 82
query43 362 376 310 310
query44 1316 699 698 698
query45 184 192 163 163
query46 858 961 564 564
query47 3039 3009 2940 2940
query48 289 302 216 216
query49 562 407 287 287
query50 675 281 210 210
query51 10673 10510 10603 10510
query52 77 78 68 68
query53 183 206 151 151
query54 229 192 208 192
query55 69 66 65 65
query56 238 214 193 193
query57 1600 1576 1577 1576
query58 272 256 246 246
query59 2356 2392 2144 2144
query60 269 236 223 223
query61 150 148 152 148
query62 394 349 291 291
query63 197 170 171 170
query64 2851 1035 873 873
query65 3452 3383 3385 3383
query66 1789 425 313 313
query67 20319 20179 20068 20068
query68 2980 1000 621 621
query69 391 308 264 264
query70 919 842 849 842
query71 279 229 222 222
query72 2801 2694 2336 2336
query73 524 521 297 297
query74 4590 4505 4303 4303
query75 2341 2285 1965 1965
query76 2287 1016 599 599
query77 357 409 304 304
query78 9173 9040 8492 8492
query79 1030 907 519 519
query80 534 413 337 337
query81 529 325 269 269
query82 788 141 116 116
query83 306 197 177 177
query84 303 118 104 104
query85 788 462 369 369
query86 317 242 228 228
query87 2015 1984 1884 1884
query88 3712 2740 2707 2707
query89 338 298 253 253
query90 1895 194 183 183
query91 157 147 122 122
query92 98 86 93 86
query93 1014 999 569 569
query94 459 287 298 287
query95 592 346 385 346
query96 683 521 235 235
query97 2509 2428 2294 2294
query98 157 145 144 144
query99 777 782 672 672
Total cold run time: 234796 ms
Total hot run time: 152533 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256817233
##########
fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java:
##########
@@ -277,7 +279,7 @@ public List<FileEntry> listFiles(Location location) throws IOException {
}
private static boolean containsGlob(String path) {
- return path.contains("*") || path.contains("?") || path.contains("[");
+ return path.contains("*") || path.contains("?") || path.contains("[") || path.contains("{");
Review Comment:
[P1] Preserve literal directory listings when enabling brace globs
For an unpartitioned Hive table at `hdfs://nn/warehouse/table_{prod}` with `hive.staging_dir=.doris_staging`, `INSERT OVERWRITE` calls `HiveConnectorTransaction.deleteTargetPathContents`, which uses `listFiles(targetPath)` to remove old files before moving staged files into the table. This new `{` check sends that literal directory through `globStatus`: [Hadoop interprets `{prod}` as a group](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GlobPattern.java#L102-L114), matching `table_prod`, and [returns an empty array when it finds no match](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L355-L377). Cleanup therefore succeeds without deleting old files, and the overwrite leaves old and new rows together. The previous direct listing read the literal directory correctly. Preserve literal-directory behavior for the
se callers while retaining brace-glob discovery, and cover a relative-staging overwrite into a directory containing braces.
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -794,7 +799,8 @@ private List<ConnectorPartitionInfo> collectPartitions(HudiTableHandle handle, b
// HoodieTableMetaClient built just to read a timeline zone - plus a fallback that
// would have published epoch 0 as a partition's last-modified time for any
// non-numeric pin, the day it became reachable.
- return buildPartitionInfos(hmsNames, partKeyNames, latestInstantMillis(handle));
+ return buildPartitionInfos(hmsPartitionPaths(handle, hmsNames),
Review Comment:
[P2] Preserve literal percent values in unencoded Hudi partitions
With `use_hive_sync_partition=true`, [Hudi's default `hoodie.datasource.write.partitionpath.urlencode=false`](https://github.com/apache/hudi/blob/release-1.0.2/hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorOptions.java#L33-L37) permits the literal STRING value `%3A` at `s3://b/t/dt=%3A`. [The Hive-style sync extractor retains `%3A`](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveStylePartitionValueExtractor.java#L32-L39), so HMS exposes `dt=%253A`. Previously `buildPartitionInfos` decoded that HMS name once and published `%3A`; the new location substitution supplies `dt=%3A`, which `parsePartitionValues` decodes into `:`. Consequently `partition_values()` now reports the wrong logical value and `SHOW PARTITIONS` changes the name; separate raw `:` and `%3A` partitions can also collapse in name-keyed consumers. Preserve the table's encoding contract when deriving logical values from physical p
aths. Add a raw `%3A` location case alongside the existing encoded `%253A` case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](external) Fix connector dependency isolation and partition discovery [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68998:
URL: https://github.com/apache/doris/pull/68998#issuecomment-6149018551
<details>
<summary>ClickBench: <b>Total hot run time: 24.61 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3039f6f48862c5ba02ef6ab5bb486daefb9c802, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.26 0.12 0.12
query4 1.61 0.12 0.13
query5 0.27 0.24 0.25
query6 1.19 0.69 0.68
query7 0.04 0.01 0.00
query8 0.05 0.04 0.04
query9 0.43 0.35 0.36
query10 0.61 0.56 0.56
query11 0.20 0.14 0.15
query12 0.18 0.15 0.15
query13 0.49 0.49 0.52
query14 0.99 0.96 0.96
query15 0.63 0.60 0.63
query16 0.35 0.35 0.35
query17 1.09 1.11 1.15
query18 0.23 0.21 0.20
query19 2.12 2.01 2.03
query20 0.02 0.01 0.02
query21 15.49 0.25 0.14
query22 4.40 0.06 0.05
query23 16.25 0.31 0.12
query24 3.10 0.62 0.50
query25 0.11 0.06 0.06
query26 1.81 0.24 0.14
query27 0.05 0.04 0.05
query28 2.96 0.66 0.42
query29 12.49 4.14 3.27
query30 0.28 0.14 0.13
query31 2.78 0.59 0.34
query32 3.23 0.61 0.52
query33 3.21 3.20 3.18
query34 15.60 3.97 3.38
query35 3.34 3.34 3.38
query36 0.55 0.45 0.47
query37 0.09 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.03 0.04
query40 0.17 0.14 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 97.03 s
Total hot run time: 24.61 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org