You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "TsukiokaKogane (via GitHub)" <gi...@apache.org> on 2026/02/05 15:57:09 UTC
[PR] [Feature] doris cross-cluster insert [doris]
TsukiokaKogane opened a new pull request, #60557:
URL: https://github.com/apache/doris/pull/60557
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #60295
Problem Summary:
1. support insert & insert overwrite on remote doris catalog
2. support temporary partitions on remote doris catalog
3. fix remote doris catalog rpc timeout unit
4. **DO NOT** support txn insert & group commit
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869898775
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/dce833ba92cf42dfecba11670310bb03dcf31394_dce833ba92cf42dfecba11670310bb03dcf31394_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/dce833ba92cf42dfecba11670310bb03dcf31394_dce833ba92cf42dfecba11670310bb03dcf31394_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.35% (1795/2262) |
| Line Coverage | 64.88% (31968/49269) |
| Region Coverage | 65.60% (15956/24323) |
| Branch Coverage | 56.12% (8482/15114) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3871165273
<details>
<summary>TPC-H: <b>Total hot run time: 30363 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef0c94e649ed972209a13ed67e7e65fa49ea4dc9, data reload: false
------ Round 1 ----------------------------------
q1 17591 4505 4291 4291
q2 2034 341 254 254
q3 10165 1262 733 733
q4 10201 767 311 311
q5 7540 2226 1918 1918
q6 203 185 157 157
q7 879 740 611 611
q8 9268 1390 1117 1117
q9 4666 4636 4614 4614
q10 6772 1932 1567 1567
q11 495 309 280 280
q12 333 375 227 227
q13 17757 4049 3278 3278
q14 231 240 209 209
q15 884 814 815 814
q16 684 656 625 625
q17 690 798 550 550
q18 6620 5960 5720 5720
q19 1237 985 627 627
q20 498 490 402 402
q21 2699 1848 1781 1781
q22 356 327 277 277
Total cold run time: 101803 ms
Total hot run time: 30363 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4369 4306 4359 4306
q2 265 347 264 264
q3 2105 2712 2197 2197
q4 1370 1750 1286 1286
q5 4336 4160 4249 4160
q6 214 181 134 134
q7 1857 1765 1648 1648
q8 2463 2703 2453 2453
q9 7680 7711 7461 7461
q10 2900 3055 2692 2692
q11 538 469 459 459
q12 722 770 632 632
q13 3958 4401 3503 3503
q14 274 297 280 280
q15 1004 838 786 786
q16 693 726 677 677
q17 1151 1293 1348 1293
q18 8109 8038 8014 8014
q19 855 896 901 896
q20 2096 2149 2024 2024
q21 5148 4510 4381 4381
q22 537 549 501 501
Total cold run time: 52644 ms
Total hot run time: 50047 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957225068
run build all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957301899
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/197a756ad3f60301963dce8f4d370fe57420930e_197a756ad3f60301963dce8f4d370fe57420930e_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/197a756ad3f60301963dce8f4d370fe57420930e_197a756ad3f60301963dce8f4d370fe57420930e_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.33% (1796/2264) |
| Line Coverage | 64.83% (32020/49393) |
| Region Coverage | 65.50% (15970/24383) |
| Branch Coverage | 56.00% (8494/15168) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3899467321
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (16/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.80% (26066/36304) |
| Line Coverage | 54.47% (272895/501013) |
| Region Coverage | 51.97% (227366/437508) |
| Branch Coverage | 53.44% (97618/182673) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2781524984
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java:
##########
@@ -173,42 +301,462 @@ public RemoteOlapTable getOlapTable(String dbName, String table, long tableId, L
removedPartitions = new ArrayList<>();
}
remoteOlapTable.rebuildPartitions(partitions, updatedPartitions, removedPartitions);
+ // rebuild temp partitions
+ if (result.isSetUpdatedTempPartitions() && result.getUpdatedTempPartitionsSize() > 0) {
+ updatedPartitions = new ArrayList<>(result.getUpdatedTempPartitionsSize());
+ for (ByteBuffer buffer : result.getUpdatedTempPartitions()) {
+ try (ByteArrayInputStream in =
+ new ByteArrayInputStream(buffer.array(), buffer.position(), buffer.remaining());
+ DataInputStream dataInputStream = new DataInputStream(in)) {
+ String partitionStr = Text.readString(dataInputStream);
+ Partition partition = GsonUtils.GSON.fromJson(partitionStr, Partition.class);
+ updatedPartitions.add(partition);
+ }
+ }
+ }
+ removedPartitions = result.getRemovedTempPartitions();
+ if (removedPartitions == null) {
+ removedPartitions = new ArrayList<>();
+ }
+ remoteOlapTable.rebuildTempPartitions(tempPartitions, updatedPartitions, removedPartitions);
return remoteOlapTable;
- }, msg, timeout);
+ }, msg, timeoutMs);
+ }
+
+ public TBeginRemoteTxnResult beginRemoteTxn(TBeginRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to begin remote txn from remote doris:%s, label=%s", name,
+ request.getLabel());
+ long startTime = System.currentTimeMillis();
+ TBeginRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.beginRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("begin remote txn for catalog {} failed, cost={}ms", name, costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("begin remote txn for catalog {} finished, cost={}ms, statusCode={}",
+ name, costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TCommitRemoteTxnResult commitRemoteTxn(TCommitRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to commit remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TCommitRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.commitRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getInsertVisibleTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("commit remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("commit remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TAbortRemoteTxnResult abortRemoteTxn(TAbortRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to abort remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TAbortRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.abortRemoteTxn(request), msg, timeoutMs);
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("abort remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("abort remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
}
private interface ThriftCall<T> {
public T call(FrontendService.Client client) throws Exception;
}
- private static class MasterResult<T> {
- boolean isMaster = true;
- T result;
- boolean hasError = false;
- String errorMsg;
+ public TNetworkAddress getMasterAddress() throws RuntimeException {
+ TMasterAddressRequest request = new TMasterAddressRequest();
+ request.setUser(user);
+ request.setPasswd(password);
+ long startTime = System.currentTimeMillis();
+ TMasterAddressResult result;
+ try {
+ result = masterCallWithRetry(client -> client.getMasterAddress(request),
+ "failed to get master address from remote doris:" + name, timeoutMs);
+ if (result.isSetMasterAddress()) {
+ master = result.getMasterAddress();
+ }
+ if (result.getStatus().getStatusCode() != TStatusCode.OK) {
+ throw new RuntimeException(result.getStatus().getErrorMsgs().get(0));
+ }
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("get master address for catalog {} failed, cost={}ms", name, costMs, e);
+ throw new RuntimeException(e.getMessage());
Review Comment:
`Util.getRootCauseMessage()` this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3871261736
# BE UT Coverage Report
Increment line coverage `13.33% (2/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.72% (19445/36883) |
| Line Coverage | 36.21% (181027/499913) |
| Region Coverage | 32.60% (140562/431180) |
| Branch Coverage | 33.63% (60883/181057) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858288115
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858433738
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9f3938347c127a752b04e13d9f40db00f84820c1_9f3938347c127a752b04e13d9f40db00f84820c1_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9f3938347c127a752b04e13d9f40db00f84820c1_9f3938347c127a752b04e13d9f40db00f84820c1_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.29% (1792/2260) |
| Line Coverage | 64.75% (31831/49158) |
| Region Coverage | 65.42% (15884/24280) |
| Branch Coverage | 55.97% (8439/15078) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957469772
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3875411041
# FE Regression Coverage Report
Increment line coverage ` 3.45% (44/1274)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869179053
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0c8af71fa683293df3e943ebd237bb3029c468ca_0c8af71fa683293df3e943ebd237bb3029c468ca_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0c8af71fa683293df3e943ebd237bb3029c468ca_0c8af71fa683293df3e943ebd237bb3029c468ca_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.35% (1795/2262) |
| Line Coverage | 64.87% (31962/49269) |
| Region Coverage | 65.53% (15940/24323) |
| Branch Coverage | 56.09% (8478/15114) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3897916289
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3898556345
<details>
<summary>TPC-H: <b>Total hot run time: 29353 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee6a26df6cc55709ed3e058d2294a831dc77e153, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17623 4479 4273 4273
q2 q3 10670 783 518 518
q4 4734 367 254 254
q5 8090 1203 1007 1007
q6 255 176 146 146
q7 806 852 663 663
q8 10655 1487 1333 1333
q9 6806 4820 4815 4815
q10 6866 1896 1638 1638
q11 442 250 254 250
q12 752 574 461 461
q13 17809 4306 3426 3426
q14 237 230 220 220
q15 935 798 797 797
q16 756 729 687 687
q17 858 898 418 418
q18 6234 5495 5406 5406
q19 1229 1039 913 913
q20 612 623 427 427
q21 4589 1921 1447 1447
q22 373 293 254 254
Total cold run time: 101331 ms
Total hot run time: 29353 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4682 4540 4572 4540
q2 q3 1814 2205 1808 1808
q4 892 1172 793 793
q5 4066 4628 4764 4628
q6 187 174 136 136
q7 1806 1640 1533 1533
q8 2604 2711 2551 2551
q9 7581 7388 7322 7322
q10 2626 2825 2673 2673
q11 522 443 418 418
q12 494 593 447 447
q13 3922 4413 3638 3638
q14 279 291 284 284
q15 870 831 797 797
q16 738 784 738 738
q17 1218 1557 1302 1302
q18 7022 6898 6675 6675
q19 897 881 883 881
q20 2171 2292 2056 2056
q21 4121 3448 3336 3336
q22 478 454 404 404
Total cold run time: 48990 ms
Total hot run time: 46960 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3899065702
# BE UT Coverage Report
Increment line coverage `4.17% (1/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ee6a26df6cc55709ed3e058d2294a831dc77e153_ee6a26df6cc55709ed3e058d2294a831dc77e153/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ee6a26df6cc55709ed3e058d2294a831dc77e153_ee6a26df6cc55709ed3e058d2294a831dc77e153/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.69% (19523/37051) |
| Line Coverage | 36.24% (182023/502262) |
| Region Coverage | 32.60% (141214/433131) |
| Branch Coverage | 33.64% (61216/181969) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2780986298
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -1662,13 +1662,178 @@ struct TGetOlapTableMetaRequest {
5: required i64 table_id
6: optional i32 version // todo serialize according to the version
7: optional list<TPartitionMeta> partitions // client owned partition meta
+ 8: optional list<TPartitionMeta> temp_partitions // client owned partition meta
}
struct TGetOlapTableMetaResult {
1: required Status.TStatus status
2: required binary table_meta
3: optional list<binary> updated_partitions
4: optional list<i64> removed_partitions
+ 5: optional list<binary> updated_temp_partitions
+ 6: optional list<i64> removed_temp_partitions
+}
+
+// Remote transaction request and Result definitions for cross-cluster export.
+// These structs are used by beginRemoteTxn/commitRemoteTxn/abortRemoteTxn RPCs.
+struct TBeginRemoteTxnRequest {
+ 1: required string user
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3870794958
be.INFO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858753986
<details>
<summary>TPC-H: <b>Total hot run time: 31323 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f3938347c127a752b04e13d9f40db00f84820c1, data reload: false
------ Round 1 ----------------------------------
q1 17617 4508 4290 4290
q2 2054 355 238 238
q3 10153 1364 770 770
q4 10211 824 325 325
q5 7499 2248 1990 1990
q6 193 178 147 147
q7 909 751 600 600
q8 9262 1405 1161 1161
q9 4847 4686 4653 4653
q10 6857 1960 1548 1548
q11 538 322 304 304
q12 336 387 230 230
q13 17778 4095 3252 3252
q14 241 234 218 218
q15 881 817 798 798
q16 672 681 642 642
q17 708 861 520 520
q18 6624 6218 6365 6218
q19 1201 1083 688 688
q20 575 545 421 421
q21 2924 2019 2068 2019
q22 380 335 291 291
Total cold run time: 102460 ms
Total hot run time: 31323 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4618 4530 4678 4530
q2 264 342 272 272
q3 2466 2879 2389 2389
q4 1520 1937 1503 1503
q5 4636 4691 4697 4691
q6 228 186 139 139
q7 1953 1882 1745 1745
q8 2553 2367 2398 2367
q9 7691 7717 7420 7420
q10 2944 3124 2614 2614
q11 540 482 468 468
q12 675 730 617 617
q13 3818 4335 3613 3613
q14 290 296 269 269
q15 825 775 782 775
q16 661 690 636 636
q17 1096 1274 1285 1274
q18 7534 7362 7433 7362
q19 897 847 841 841
q20 1977 2104 1878 1878
q21 4597 4245 4078 4078
q22 560 544 506 506
Total cold run time: 52343 ms
Total hot run time: 49987 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858209545
# BE UT Coverage Report
Increment line coverage `13.33% (2/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/97168897275f4683257ec006a83a0f0b792a824d_97168897275f4683257ec006a83a0f0b792a824d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/97168897275f4683257ec006a83a0f0b792a824d_97168897275f4683257ec006a83a0f0b792a824d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.56% (19376/36862) |
| Line Coverage | 36.06% (180041/499291) |
| Region Coverage | 32.42% (139548/430471) |
| Branch Coverage | 33.42% (60437/180846) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2780986906
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -1662,13 +1662,178 @@ struct TGetOlapTableMetaRequest {
5: required i64 table_id
6: optional i32 version // todo serialize according to the version
7: optional list<TPartitionMeta> partitions // client owned partition meta
+ 8: optional list<TPartitionMeta> temp_partitions // client owned partition meta
}
struct TGetOlapTableMetaResult {
1: required Status.TStatus status
2: required binary table_meta
3: optional list<binary> updated_partitions
4: optional list<i64> removed_partitions
+ 5: optional list<binary> updated_temp_partitions
+ 6: optional list<i64> removed_temp_partitions
+}
+
+// Remote transaction request and Result definitions for cross-cluster export.
+// These structs are used by beginRemoteTxn/commitRemoteTxn/abortRemoteTxn RPCs.
+struct TBeginRemoteTxnRequest {
+ 1: required string user
+ 2: required string passwd
+ 3: optional string token
+ 4: required string db
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869773306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3857538215
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869558473
<details>
<summary>ClickBench: <b>Total hot run time: 28.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c8af71fa683293df3e943ebd237bb3029c468ca, data reload: false
query1 0.05 0.05 0.05
query2 0.14 0.07 0.07
query3 0.32 0.08 0.08
query4 1.61 0.10 0.09
query5 0.27 0.24 0.24
query6 1.14 0.64 0.64
query7 0.03 0.03 0.03
query8 0.07 0.05 0.07
query9 0.58 0.51 0.50
query10 0.54 0.55 0.55
query11 0.26 0.13 0.14
query12 0.27 0.15 0.15
query13 0.62 0.61 0.61
query14 0.98 0.97 0.98
query15 0.91 0.82 0.81
query16 0.39 0.40 0.40
query17 1.06 0.99 0.99
query18 0.24 0.25 0.23
query19 2.00 1.87 1.86
query20 0.02 0.02 0.01
query21 15.40 0.35 0.30
query22 4.93 0.13 0.12
query23 15.38 0.45 0.29
query24 2.36 0.59 0.41
query25 0.11 0.11 0.12
query26 0.20 0.18 0.19
query27 0.11 0.12 0.11
query28 3.58 1.17 0.98
query29 12.51 4.07 3.29
query30 0.33 0.12 0.11
query31 2.80 0.68 0.44
query32 3.24 0.63 0.50
query33 3.00 3.03 3.13
query34 16.33 5.09 4.45
query35 4.49 4.44 4.43
query36 0.62 0.49 0.49
query37 0.31 0.08 0.08
query38 0.28 0.06 0.06
query39 0.08 0.05 0.05
query40 0.21 0.17 0.17
query41 0.13 0.07 0.08
query42 0.09 0.05 0.05
query43 0.06 0.06 0.05
Total cold run time: 98.05 s
Total hot run time: 28.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3871037940
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.35% (1795/2262) |
| Line Coverage | 64.86% (31956/49269) |
| Region Coverage | 65.53% (15940/24323) |
| Branch Coverage | 56.08% (8476/15114) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3870613458
# FE UT Coverage Report
Increment line coverage ` 1.89% (24/1272)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_dce833ba92cf42dfecba11670310bb03dcf31394/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_dce833ba92cf42dfecba11670310bb03dcf31394/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3875062481
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3900921003
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901273248
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (16/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.71% (26032/36304) |
| Line Coverage | 54.38% (272440/501013) |
| Region Coverage | 51.85% (226842/437508) |
| Branch Coverage | 53.34% (97432/182673) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901649315
# FE Regression Coverage Report
Increment line coverage ` 28.43% (356/1252)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3898607593
<details>
<summary>TPC-DS: <b>Total hot run time: 183506 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee6a26df6cc55709ed3e058d2294a831dc77e153, data reload: false
query5 4772 641 529 529
query6 337 223 201 201
query7 4229 469 281 281
query8 345 250 237 237
query9 8760 2732 2706 2706
query10 528 384 357 357
query11 16961 17432 17089 17089
query12 185 132 122 122
query13 1388 558 365 365
query14 6921 3264 3202 3202
query14_1 3000 2850 2889 2850
query15 218 200 196 196
query16 1069 498 472 472
query17 1609 772 624 624
query18 2751 437 346 346
query19 207 200 176 176
query20 132 125 125 125
query21 225 138 112 112
query22 4914 4872 5071 4872
query23 17260 16893 16583 16583
query23_1 16735 16653 16621 16621
query24 7086 1598 1212 1212
query24_1 1219 1262 1230 1230
query25 524 440 447 440
query26 1234 259 147 147
query27 2760 465 282 282
query28 4472 1866 1852 1852
query29 786 555 465 465
query30 312 248 211 211
query31 896 715 648 648
query32 79 76 71 71
query33 508 330 275 275
query34 921 897 573 573
query35 650 692 602 602
query36 1086 1141 999 999
query37 130 93 84 84
query38 2985 3007 2845 2845
query39 883 877 838 838
query39_1 823 828 826 826
query40 231 150 140 140
query41 62 62 58 58
query42 104 101 101 101
query43 394 383 351 351
query44
query45 195 187 180 180
query46 885 976 600 600
query47 2154 2197 2080 2080
query48 307 329 234 234
query49 628 470 377 377
query50 686 283 217 217
query51 4074 4091 4030 4030
query52 108 107 94 94
query53 285 333 285 285
query54 296 264 261 261
query55 86 84 81 81
query56 303 304 315 304
query57 1402 1342 1267 1267
query58 285 276 270 270
query59 2602 2689 2494 2494
query60 334 336 319 319
query61 155 148 152 148
query62 622 591 544 544
query63 316 277 277 277
query64 4779 1248 985 985
query65
query66 1403 465 380 380
query67 16545 16405 16280 16280
query68
query69 400 311 303 303
query70 1007 984 968 968
query71 344 309 308 308
query72 2943 2917 2496 2496
query73 540 544 313 313
query74 10007 9892 9789 9789
query75 2832 2718 2455 2455
query76 2288 1045 668 668
query77 356 395 315 315
query78 11247 11470 10701 10701
query79 2538 809 604 604
query80 1786 620 536 536
query81 563 286 256 256
query82 1003 150 112 112
query83 356 255 244 244
query84 248 115 95 95
query85 991 494 431 431
query86 434 326 294 294
query87 3124 3115 2989 2989
query88 3557 2669 2638 2638
query89 432 365 339 339
query90 1965 169 170 169
query91 160 156 132 132
query92 79 75 73 73
query93 1440 832 506 506
query94 633 305 298 298
query95 603 399 313 313
query96 644 521 228 228
query97 2521 2472 2420 2420
query98 222 224 217 217
query99 1026 998 907 907
Total cold run time: 256853 ms
Total hot run time: 183506 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3958080480
<details>
<summary>TPC-H: <b>Total hot run time: 28827 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d552c156312988d13a7e1dc37a84672169ae2f57, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17636 4546 4340 4340
q2 q3 10630 838 547 547
q4 4680 351 255 255
q5 7545 1202 1044 1044
q6 175 177 148 148
q7 783 843 672 672
q8 9288 1507 1333 1333
q9 4880 4809 4784 4784
q10 6836 1884 1642 1642
q11 440 248 245 245
q12 717 572 478 478
q13 17765 4245 3419 3419
q14 228 230 221 221
q15 959 794 788 788
q16 764 735 666 666
q17 721 864 428 428
q18 6134 5351 5203 5203
q19 1132 987 601 601
q20 507 486 388 388
q21 4506 1867 1389 1389
q22 348 280 236 236
Total cold run time: 96674 ms
Total hot run time: 28827 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4443 4356 4354 4354
q2 q3 1788 2189 1710 1710
q4 855 1160 740 740
q5 4073 4292 4383 4292
q6 181 173 141 141
q7 1728 1603 1476 1476
q8 2421 2643 2517 2517
q9 7616 7503 7468 7468
q10 2623 2851 2423 2423
q11 507 436 413 413
q12 495 585 468 468
q13 4106 4517 3630 3630
q14 348 313 297 297
q15 861 785 822 785
q16 719 766 717 717
q17 1191 1527 1326 1326
q18 7114 6827 6781 6781
q19 889 939 958 939
q20 2097 2196 2045 2045
q21 4023 3486 3555 3486
q22 436 437 376 376
Total cold run time: 48514 ms
Total hot run time: 46384 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3960132055
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (16/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_d552c156312988d13a7e1dc37a84672169ae2f57_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_d552c156312988d13a7e1dc37a84672169ae2f57_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.55% (26133/36525) |
| Line Coverage | 54.30% (273664/503995) |
| Region Coverage | 51.57% (227257/440710) |
| Branch Coverage | 53.04% (97798/184398) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3959308396
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3857677814
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/97168897275f4683257ec006a83a0f0b792a824d_97168897275f4683257ec006a83a0f0b792a824d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/97168897275f4683257ec006a83a0f0b792a824d_97168897275f4683257ec006a83a0f0b792a824d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.29% (1792/2260) |
| Line Coverage | 64.74% (31825/49158) |
| Region Coverage | 65.41% (15882/24280) |
| Branch Coverage | 55.94% (8435/15078) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858008635
<details>
<summary>ClickBench: <b>Total hot run time: 28.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97168897275f4683257ec006a83a0f0b792a824d, data reload: false
query1 0.05 0.05 0.05
query2 0.13 0.07 0.07
query3 0.32 0.07 0.07
query4 1.61 0.10 0.10
query5 0.25 0.25 0.24
query6 1.15 0.66 0.64
query7 0.03 0.02 0.02
query8 0.08 0.06 0.06
query9 0.59 0.50 0.49
query10 0.54 0.56 0.54
query11 0.27 0.13 0.14
query12 0.27 0.14 0.14
query13 0.64 0.63 0.60
query14 0.98 0.98 0.98
query15 0.94 0.83 0.83
query16 0.39 0.40 0.39
query17 1.06 1.04 1.05
query18 0.26 0.23 0.23
query19 1.84 1.83 1.79
query20 0.02 0.01 0.02
query21 15.38 0.34 0.29
query22 4.98 0.11 0.12
query23 15.36 0.45 0.28
query24 2.30 0.56 0.39
query25 0.11 0.11 0.11
query26 0.19 0.18 0.18
query27 0.10 0.10 0.10
query28 3.62 1.14 0.98
query29 12.52 4.12 3.31
query30 0.34 0.12 0.11
query31 2.79 0.70 0.44
query32 3.24 0.64 0.51
query33 2.95 3.11 3.04
query34 16.30 5.07 4.46
query35 4.47 4.47 4.48
query36 0.62 0.50 0.50
query37 0.30 0.10 0.09
query38 0.26 0.06 0.06
query39 0.08 0.05 0.04
query40 0.22 0.17 0.18
query41 0.13 0.08 0.06
query42 0.08 0.05 0.05
query43 0.06 0.06 0.06
Total cold run time: 97.82 s
Total hot run time: 28.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3859914530
# BE Regression && UT Coverage Report
Increment line coverage `73.33% (11/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_9f3938347c127a752b04e13d9f40db00f84820c1_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_9f3938347c127a752b04e13d9f40db00f84820c1_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.72% (25930/36156) |
| Line Coverage | 54.39% (271270/498758) |
| Region Coverage | 51.96% (226378/435653) |
| Branch Coverage | 53.36% (97020/181809) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869299538
# FE UT Coverage Report
Increment line coverage ` 2.02% (24/1189)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_0c8af71fa683293df3e943ebd237bb3029c468ca/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_0c8af71fa683293df3e943ebd237bb3029c468ca/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869111759
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2780638429
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java:
##########
@@ -173,42 +301,462 @@ public RemoteOlapTable getOlapTable(String dbName, String table, long tableId, L
removedPartitions = new ArrayList<>();
}
remoteOlapTable.rebuildPartitions(partitions, updatedPartitions, removedPartitions);
+ // rebuild temp partitions
+ if (result.isSetUpdatedTempPartitions() && result.getUpdatedTempPartitionsSize() > 0) {
+ updatedPartitions = new ArrayList<>(result.getUpdatedTempPartitionsSize());
+ for (ByteBuffer buffer : result.getUpdatedTempPartitions()) {
+ try (ByteArrayInputStream in =
+ new ByteArrayInputStream(buffer.array(), buffer.position(), buffer.remaining());
+ DataInputStream dataInputStream = new DataInputStream(in)) {
+ String partitionStr = Text.readString(dataInputStream);
+ Partition partition = GsonUtils.GSON.fromJson(partitionStr, Partition.class);
+ updatedPartitions.add(partition);
+ }
+ }
+ }
+ removedPartitions = result.getRemovedTempPartitions();
+ if (removedPartitions == null) {
+ removedPartitions = new ArrayList<>();
+ }
+ remoteOlapTable.rebuildTempPartitions(tempPartitions, updatedPartitions, removedPartitions);
return remoteOlapTable;
- }, msg, timeout);
+ }, msg, timeoutMs);
+ }
+
+ public TBeginRemoteTxnResult beginRemoteTxn(TBeginRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to begin remote txn from remote doris:%s, label=%s", name,
+ request.getLabel());
+ long startTime = System.currentTimeMillis();
+ TBeginRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.beginRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("begin remote txn for catalog {} failed, cost={}ms", name, costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("begin remote txn for catalog {} finished, cost={}ms, statusCode={}",
+ name, costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TCommitRemoteTxnResult commitRemoteTxn(TCommitRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to commit remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TCommitRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.commitRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getInsertVisibleTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("commit remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("commit remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TAbortRemoteTxnResult abortRemoteTxn(TAbortRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to abort remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TAbortRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.abortRemoteTxn(request), msg, timeoutMs);
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("abort remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("abort remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
}
private interface ThriftCall<T> {
public T call(FrontendService.Client client) throws Exception;
}
- private static class MasterResult<T> {
- boolean isMaster = true;
- T result;
- boolean hasError = false;
- String errorMsg;
+ public TNetworkAddress getMasterAddress() throws RuntimeException {
+ TMasterAddressRequest request = new TMasterAddressRequest();
+ request.setUser(user);
+ request.setPasswd(password);
+ long startTime = System.currentTimeMillis();
+ TMasterAddressResult result;
+ try {
+ result = masterCallWithRetry(client -> client.getMasterAddress(request),
+ "failed to get master address from remote doris:" + name, timeoutMs);
+ if (result.isSetMasterAddress()) {
+ master = result.getMasterAddress();
+ }
+ if (result.getStatus().getStatusCode() != TStatusCode.OK) {
+ throw new RuntimeException(result.getStatus().getErrorMsgs().get(0));
+ }
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("get master address for catalog {} failed, cost={}ms", name, costMs, e);
+ throw new RuntimeException(e.getMessage());
Review Comment:
do you mean getRootCauseStack?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901107898
# FE Regression Coverage Report
Increment line coverage ` 28.51% (357/1252)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901389596
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957856029
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3958757262
# BE UT Coverage Report
Increment line coverage `4.17% (1/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/d552c156312988d13a7e1dc37a84672169ae2f57_d552c156312988d13a7e1dc37a84672169ae2f57/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/d552c156312988d13a7e1dc37a84672169ae2f57_d552c156312988d13a7e1dc37a84672169ae2f57/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.56% (19602/37292) |
| Line Coverage | 36.17% (182853/505495) |
| Region Coverage | 32.49% (141820/436542) |
| Branch Coverage | 33.45% (61480/183812) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3900870104
run p0 cloud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901285578
# FE Regression Coverage Report
Increment line coverage ` 28.43% (356/1252)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3898076989
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ee6a26df6cc55709ed3e058d2294a831dc77e153_ee6a26df6cc55709ed3e058d2294a831dc77e153_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ee6a26df6cc55709ed3e058d2294a831dc77e153_ee6a26df6cc55709ed3e058d2294a831dc77e153_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.33% (1796/2264) |
| Line Coverage | 64.83% (32009/49372) |
| Region Coverage | 65.51% (15971/24378) |
| Branch Coverage | 56.03% (8496/15164) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869499458
<details>
<summary>TPC-H: <b>Total hot run time: 30323 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c8af71fa683293df3e943ebd237bb3029c468ca, data reload: false
------ Round 1 ----------------------------------
q1 17633 4474 4351 4351
q2 2042 347 228 228
q3 10642 1375 735 735
q4 10370 779 315 315
q5 9658 2230 1932 1932
q6 238 183 143 143
q7 896 735 613 613
q8 9272 1398 1171 1171
q9 4850 4679 4637 4637
q10 6852 1985 1557 1557
q11 515 320 288 288
q12 404 381 223 223
q13 17793 4060 3237 3237
q14 245 245 212 212
q15 905 828 802 802
q16 683 679 622 622
q17 924 815 530 530
q18 7147 5838 5631 5631
q19 1251 997 619 619
q20 499 505 386 386
q21 2585 1866 1810 1810
q22 368 324 281 281
Total cold run time: 105772 ms
Total hot run time: 30323 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4469 4385 4366 4366
q2 275 332 257 257
q3 2076 2704 2231 2231
q4 1345 1756 1301 1301
q5 4287 4200 4200 4200
q6 219 182 137 137
q7 1853 1816 2087 1816
q8 2669 2417 2380 2380
q9 7576 7570 7535 7535
q10 2878 3171 2621 2621
q11 546 480 481 480
q12 742 759 596 596
q13 3898 4551 3681 3681
q14 289 296 298 296
q15 841 796 806 796
q16 675 829 858 829
q17 1187 1279 1323 1279
q18 8252 7949 7857 7857
q19 905 918 872 872
q20 2074 2121 2006 2006
q21 4709 4380 4146 4146
q22 582 546 509 509
Total cold run time: 52347 ms
Total hot run time: 50191 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869069061
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3871238375
<details>
<summary>ClickBench: <b>Total hot run time: 28.03 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef0c94e649ed972209a13ed67e7e65fa49ea4dc9, data reload: false
query1 0.06 0.05 0.04
query2 0.13 0.07 0.07
query3 0.31 0.08 0.07
query4 1.61 0.10 0.10
query5 0.27 0.25 0.27
query6 1.14 0.65 0.64
query7 0.03 0.03 0.03
query8 0.07 0.06 0.06
query9 0.58 0.50 0.48
query10 0.54 0.56 0.55
query11 0.26 0.13 0.14
query12 0.26 0.14 0.17
query13 0.61 0.62 0.59
query14 0.97 0.97 0.98
query15 0.91 0.82 0.82
query16 0.38 0.38 0.38
query17 1.05 1.08 1.02
query18 0.23 0.22 0.23
query19 1.88 1.81 1.82
query20 0.02 0.02 0.01
query21 15.39 0.33 0.31
query22 4.94 0.13 0.12
query23 15.34 0.46 0.28
query24 2.24 0.56 0.38
query25 0.11 0.10 0.10
query26 0.19 0.18 0.18
query27 0.10 0.10 0.11
query28 3.51 1.17 0.98
query29 12.50 4.02 3.24
query30 0.32 0.12 0.11
query31 2.80 0.66 0.43
query32 3.24 0.63 0.50
query33 3.07 3.01 3.08
query34 16.33 5.04 4.45
query35 4.49 4.50 4.46
query36 0.61 0.50 0.48
query37 0.30 0.09 0.08
query38 0.28 0.06 0.06
query39 0.08 0.05 0.05
query40 0.19 0.18 0.17
query41 0.13 0.07 0.06
query42 0.08 0.06 0.05
query43 0.07 0.05 0.05
Total cold run time: 97.62 s
Total hot run time: 28.03 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3869710393
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957226068
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957613426
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/d552c156312988d13a7e1dc37a84672169ae2f57_d552c156312988d13a7e1dc37a84672169ae2f57_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/d552c156312988d13a7e1dc37a84672169ae2f57_d552c156312988d13a7e1dc37a84672169ae2f57_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.33% (1796/2264) |
| Line Coverage | 64.79% (32004/49393) |
| Region Coverage | 65.51% (15974/24383) |
| Branch Coverage | 55.99% (8492/15168) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2780544874
##########
regression-test/suites/external_table_p0/remote_doris/test_insert_remote_doris_as_olap_table_select.groovy:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_insert_remote_doris_as_olap_table_select", "p0,external,doris,external_docker,external_docker_doris") {
+ String remote_doris_host = context.config.otherConfigs.get("extArrowFlightSqlHost")
+ String remote_doris_arrow_port = context.config.otherConfigs.get("extArrowFlightSqlPort")
+ String remote_doris_http_port = context.config.otherConfigs.get("extArrowFlightHttpPort")
+ String remote_doris_user = context.config.otherConfigs.get("extArrowFlightSqlUser")
+ String remote_doris_psw = context.config.otherConfigs.get("extArrowFlightSqlPassword")
+ String remote_doris_thrift_port = context.config.otherConfigs.get("extFeThriftPort")
+
+ def showres = sql "show frontends";
+ remote_doris_arrow_port = showres[0][6]
+ remote_doris_http_port = showres[0][3]
+ remote_doris_thrift_port = showres[0][5]
+ log.info("show frontends log = ${showres}, arrow: ${remote_doris_arrow_port}, http: ${remote_doris_http_port}, thrift: ${remote_doris_thrift_port}")
+
+ def showres2 = sql "show backends";
+ log.info("show backends log = ${showres2}")
+
+ def db_name = "test_insert_remote_doris_as_olap_table_select_db"
+ def catalog_name = "test_insert_remote_doris_as_olap_table_select_catalog"
+
+ sql """DROP DATABASE IF EXISTS ${db_name}"""
+
+ sql """CREATE DATABASE IF NOT EXISTS ${db_name}"""
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ DROP CATALOG IF EXISTS `${catalog_name}`
+ """
+
+
+ sql """
+ CREATE CATALOG `${catalog_name}` PROPERTIES (
+ 'type' = 'doris',
+ 'fe_http_hosts' = 'http://${remote_doris_host}:${remote_doris_http_port}',
+ 'fe_arrow_hosts' = '${remote_doris_host}:${remote_doris_arrow_port}',
+ 'fe_thrift_hosts' = '${remote_doris_host}:${remote_doris_thrift_port}',
+ 'user' = '${remote_doris_user}',
+ 'password' = '${remote_doris_psw}',
+ 'use_arrow_flight' = 'false'
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 03:00:00.000', false, 127, 32767, 2147483647, 9223372036854775807, 1234567890123456789, 123.456, 123456.789, 123457, 123456789012346, 1234567890123456789012345678, '9999-12-31', '9999-12-31 23:59:59', '', '', '', [], {}, {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 04:00:00.000', true, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '2023-10-01', '2023-10-01 12:34:56', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"});
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t2` (
+ `id` datetime(3) NOT NULL,
+ `a_boolean` array<boolean> NULL,
+ `a_tinyint` array<tinyint> NULL,
+ `a_smallint` array<smallint> NULL,
+ `a_int` array<int> NULL,
+ `a_bigint` array<bigint> NULL,
+ `a_largeint` array<largeint> NULL,
+ `a_float` array<float> NULL,
+ `a_double` array<double> NULL,
+ `a_decimal9` array<decimal(9,0)> NULL,
+ `a_decimal18` array<decimal(18,0)> NULL,
+ `a_decimal32` array<decimal(32,0)> NULL,
+ `a_date` array<date> NULL,
+ `a_datetime` array<datetime> NULL,
+ `a_char` array<char(1)> NULL,
+ `a_varchar` array<varchar(65533)> NULL,
+ `a_string` array<text> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648], [-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789], [-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'], ['0000-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!'])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 02:00:00.000', [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 03:00:00.000', [false], [127], [32767], [2147483647], [9223372036854775807], [1234567890123456789], [123.456], [123456.789], [123457], [123456789012346], [1234567890123456789012345678], ['9999-12-31'], ['9999-12-31 23:59:59'], [''], [''], [''])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 04:00:00.000', [true], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], ['2023-10-01'], ['2023-10-01 12:34:56'], ['A'], ['Hello'], ['Hello, Doris!']);
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t3` (
+ `id` datetime NOT NULL,
+ `datetime_0` datetime(0) NULL,
+ `datetime_1` datetime(1) NULL,
+ `datetime_3` datetime(2) NULL,
+ `datetime_4` datetime(3) NULL,
+ `datetime_5` datetime(4) NULL,
+ `datetime_6` datetime(5) NULL,
+ `datetime_7` datetime(6) NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111');
+ """
+
+
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test select after insert also get correct data, it seems we can newly partition version
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111');
+ """
+ qt_query_after_insert """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test insert command
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_insert` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+ // test insert into
+ explain {
+ sql("insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
+ """
+ qt_after_insert_cmd """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` order by id
+ """
+ // test insert overwrite
+ explain {
+ sql("insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
Review Comment:
i create the regression test file based on remote doris query and replacing target table with remote olap catalog table, most of the session variables other than timeout should not take in effect in remote clusters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3857948948
<details>
<summary>TPC-H: <b>Total hot run time: 31471 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97168897275f4683257ec006a83a0f0b792a824d, data reload: false
------ Round 1 ----------------------------------
q1 17660 4522 4407 4407
q2 2051 349 224 224
q3 10469 1342 767 767
q4 10343 809 329 329
q5 9507 2263 1927 1927
q6 228 182 156 156
q7 883 750 590 590
q8 9266 1447 1221 1221
q9 5356 4836 4890 4836
q10 6888 1959 1582 1582
q11 503 304 290 290
q12 398 377 231 231
q13 17813 4066 3274 3274
q14 231 246 214 214
q15 924 811 804 804
q16 689 684 621 621
q17 708 767 547 547
q18 6755 6412 6330 6330
q19 1326 1004 654 654
q20 395 357 256 256
q21 2709 2004 1937 1937
q22 358 318 274 274
Total cold run time: 105460 ms
Total hot run time: 31471 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4442 4333 4374 4333
q2 257 330 265 265
q3 2137 2601 2229 2229
q4 1372 1735 1327 1327
q5 4327 4207 4182 4182
q6 216 184 140 140
q7 1884 2156 1917 1917
q8 2654 2501 2431 2431
q9 7635 7459 7620 7459
q10 2784 3063 2750 2750
q11 570 503 455 455
q12 684 752 658 658
q13 3851 4494 3582 3582
q14 305 302 288 288
q15 851 802 793 793
q16 694 720 728 720
q17 1198 1489 1385 1385
q18 8328 8094 7920 7920
q19 936 920 914 914
q20 2119 2144 1962 1962
q21 4524 4352 4109 4109
q22 544 564 504 504
Total cold run time: 52312 ms
Total hot run time: 50323 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858538946
# FE UT Coverage Report
Increment line coverage ` 2.44% (29/1189)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_9f3938347c127a752b04e13d9f40db00f84820c1/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_9f3938347c127a752b04e13d9f40db00f84820c1/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901059324
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (16/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.71% (26032/36304) |
| Line Coverage | 54.38% (272440/501013) |
| Region Coverage | 51.85% (226842/437508) |
| Branch Coverage | 53.34% (97432/182673) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2781729039
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java:
##########
@@ -173,42 +301,462 @@ public RemoteOlapTable getOlapTable(String dbName, String table, long tableId, L
removedPartitions = new ArrayList<>();
}
remoteOlapTable.rebuildPartitions(partitions, updatedPartitions, removedPartitions);
+ // rebuild temp partitions
+ if (result.isSetUpdatedTempPartitions() && result.getUpdatedTempPartitionsSize() > 0) {
+ updatedPartitions = new ArrayList<>(result.getUpdatedTempPartitionsSize());
+ for (ByteBuffer buffer : result.getUpdatedTempPartitions()) {
+ try (ByteArrayInputStream in =
+ new ByteArrayInputStream(buffer.array(), buffer.position(), buffer.remaining());
+ DataInputStream dataInputStream = new DataInputStream(in)) {
+ String partitionStr = Text.readString(dataInputStream);
+ Partition partition = GsonUtils.GSON.fromJson(partitionStr, Partition.class);
+ updatedPartitions.add(partition);
+ }
+ }
+ }
+ removedPartitions = result.getRemovedTempPartitions();
+ if (removedPartitions == null) {
+ removedPartitions = new ArrayList<>();
+ }
+ remoteOlapTable.rebuildTempPartitions(tempPartitions, updatedPartitions, removedPartitions);
return remoteOlapTable;
- }, msg, timeout);
+ }, msg, timeoutMs);
+ }
+
+ public TBeginRemoteTxnResult beginRemoteTxn(TBeginRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to begin remote txn from remote doris:%s, label=%s", name,
+ request.getLabel());
+ long startTime = System.currentTimeMillis();
+ TBeginRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.beginRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("begin remote txn for catalog {} failed, cost={}ms", name, costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("begin remote txn for catalog {} finished, cost={}ms, statusCode={}",
+ name, costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TCommitRemoteTxnResult commitRemoteTxn(TCommitRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to commit remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TCommitRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.commitRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getInsertVisibleTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("commit remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("commit remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TAbortRemoteTxnResult abortRemoteTxn(TAbortRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to abort remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TAbortRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.abortRemoteTxn(request), msg, timeoutMs);
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("abort remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("abort remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
}
private interface ThriftCall<T> {
public T call(FrontendService.Client client) throws Exception;
}
- private static class MasterResult<T> {
- boolean isMaster = true;
- T result;
- boolean hasError = false;
- String errorMsg;
+ public TNetworkAddress getMasterAddress() throws RuntimeException {
+ TMasterAddressRequest request = new TMasterAddressRequest();
+ request.setUser(user);
+ request.setPasswd(password);
+ long startTime = System.currentTimeMillis();
+ TMasterAddressResult result;
+ try {
+ result = masterCallWithRetry(client -> client.getMasterAddress(request),
+ "failed to get master address from remote doris:" + name, timeoutMs);
+ if (result.isSetMasterAddress()) {
+ master = result.getMasterAddress();
+ }
+ if (result.getStatus().getStatusCode() != TStatusCode.OK) {
+ throw new RuntimeException(result.getStatus().getErrorMsgs().get(0));
+ }
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("get master address for catalog {} failed, cost={}ms", name, costMs, e);
+ throw new RuntimeException(e.getMessage());
Review Comment:
fixed
##########
regression-test/suites/external_table_p0/remote_doris/test_insert_remote_doris_as_olap_table_select.groovy:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_insert_remote_doris_as_olap_table_select", "p0,external,doris,external_docker,external_docker_doris") {
+ String remote_doris_host = context.config.otherConfigs.get("extArrowFlightSqlHost")
+ String remote_doris_arrow_port = context.config.otherConfigs.get("extArrowFlightSqlPort")
+ String remote_doris_http_port = context.config.otherConfigs.get("extArrowFlightHttpPort")
+ String remote_doris_user = context.config.otherConfigs.get("extArrowFlightSqlUser")
+ String remote_doris_psw = context.config.otherConfigs.get("extArrowFlightSqlPassword")
+ String remote_doris_thrift_port = context.config.otherConfigs.get("extFeThriftPort")
+
+ def showres = sql "show frontends";
+ remote_doris_arrow_port = showres[0][6]
+ remote_doris_http_port = showres[0][3]
+ remote_doris_thrift_port = showres[0][5]
+ log.info("show frontends log = ${showres}, arrow: ${remote_doris_arrow_port}, http: ${remote_doris_http_port}, thrift: ${remote_doris_thrift_port}")
+
+ def showres2 = sql "show backends";
+ log.info("show backends log = ${showres2}")
+
+ def db_name = "test_insert_remote_doris_as_olap_table_select_db"
+ def catalog_name = "test_insert_remote_doris_as_olap_table_select_catalog"
+
+ sql """DROP DATABASE IF EXISTS ${db_name}"""
+
+ sql """CREATE DATABASE IF NOT EXISTS ${db_name}"""
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ DROP CATALOG IF EXISTS `${catalog_name}`
+ """
+
+
+ sql """
+ CREATE CATALOG `${catalog_name}` PROPERTIES (
+ 'type' = 'doris',
+ 'fe_http_hosts' = 'http://${remote_doris_host}:${remote_doris_http_port}',
+ 'fe_arrow_hosts' = '${remote_doris_host}:${remote_doris_arrow_port}',
+ 'fe_thrift_hosts' = '${remote_doris_host}:${remote_doris_thrift_port}',
+ 'user' = '${remote_doris_user}',
+ 'password' = '${remote_doris_psw}',
+ 'use_arrow_flight' = 'false'
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 03:00:00.000', false, 127, 32767, 2147483647, 9223372036854775807, 1234567890123456789, 123.456, 123456.789, 123457, 123456789012346, 1234567890123456789012345678, '9999-12-31', '9999-12-31 23:59:59', '', '', '', [], {}, {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 04:00:00.000', true, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '2023-10-01', '2023-10-01 12:34:56', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"});
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t2` (
+ `id` datetime(3) NOT NULL,
+ `a_boolean` array<boolean> NULL,
+ `a_tinyint` array<tinyint> NULL,
+ `a_smallint` array<smallint> NULL,
+ `a_int` array<int> NULL,
+ `a_bigint` array<bigint> NULL,
+ `a_largeint` array<largeint> NULL,
+ `a_float` array<float> NULL,
+ `a_double` array<double> NULL,
+ `a_decimal9` array<decimal(9,0)> NULL,
+ `a_decimal18` array<decimal(18,0)> NULL,
+ `a_decimal32` array<decimal(32,0)> NULL,
+ `a_date` array<date> NULL,
+ `a_datetime` array<datetime> NULL,
+ `a_char` array<char(1)> NULL,
+ `a_varchar` array<varchar(65533)> NULL,
+ `a_string` array<text> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648], [-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789], [-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'], ['0000-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!'])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 02:00:00.000', [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 03:00:00.000', [false], [127], [32767], [2147483647], [9223372036854775807], [1234567890123456789], [123.456], [123456.789], [123457], [123456789012346], [1234567890123456789012345678], ['9999-12-31'], ['9999-12-31 23:59:59'], [''], [''], [''])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 04:00:00.000', [true], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], ['2023-10-01'], ['2023-10-01 12:34:56'], ['A'], ['Hello'], ['Hello, Doris!']);
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t3` (
+ `id` datetime NOT NULL,
+ `datetime_0` datetime(0) NULL,
+ `datetime_1` datetime(1) NULL,
+ `datetime_3` datetime(2) NULL,
+ `datetime_4` datetime(3) NULL,
+ `datetime_5` datetime(4) NULL,
+ `datetime_6` datetime(5) NULL,
+ `datetime_7` datetime(6) NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111');
+ """
+
+
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test select after insert also get correct data, it seems we can newly partition version
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111');
+ """
+ qt_query_after_insert """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test insert command
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_insert` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+ // test insert into
+ explain {
+ sql("insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
+ """
+ qt_after_insert_cmd """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` order by id
+ """
+ // test insert overwrite
+ explain {
+ sql("insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3870796185
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3901634320
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (16/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.80% (26066/36304) |
| Line Coverage | 54.47% (272900/501013) |
| Region Coverage | 51.97% (227365/437508) |
| Branch Coverage | 53.44% (97612/182673) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3957855789
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3854558218
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3859968153
# FE Regression Coverage Report
Increment line coverage ` 29.44% (350/1189)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_9f3938347c127a752b04e13d9f40db00f84820c1_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_9f3938347c127a752b04e13d9f40db00f84820c1_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3858823722
<details>
<summary>ClickBench: <b>Total hot run time: 28.66 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f3938347c127a752b04e13d9f40db00f84820c1, data reload: false
query1 0.05 0.05 0.05
query2 0.13 0.07 0.07
query3 0.32 0.08 0.08
query4 1.61 0.10 0.10
query5 0.28 0.25 0.25
query6 1.15 0.65 0.65
query7 0.03 0.03 0.02
query8 0.07 0.06 0.06
query9 0.58 0.51 0.50
query10 0.55 0.56 0.54
query11 0.27 0.14 0.14
query12 0.26 0.14 0.15
query13 0.62 0.60 0.60
query14 0.98 0.99 0.98
query15 0.89 0.82 0.83
query16 0.45 0.39 0.39
query17 0.99 1.08 0.97
query18 0.25 0.23 0.23
query19 1.94 1.85 1.88
query20 0.02 0.02 0.01
query21 15.40 0.34 0.28
query22 4.92 0.11 0.12
query23 15.34 0.44 0.27
query24 2.34 0.56 0.37
query25 0.11 0.10 0.11
query26 0.19 0.18 0.19
query27 0.11 0.11 0.10
query28 3.62 1.16 0.99
query29 12.54 4.13 3.34
query30 0.33 0.13 0.11
query31 2.79 0.70 0.45
query32 3.24 0.62 0.50
query33 3.00 3.06 3.01
query34 16.27 5.10 4.42
query35 4.53 4.91 5.05
query36 0.69 0.58 0.55
query37 0.33 0.08 0.09
query38 0.28 0.06 0.06
query39 0.08 0.05 0.05
query40 0.20 0.18 0.16
query41 0.14 0.08 0.07
query42 0.09 0.05 0.05
query43 0.06 0.06 0.05
Total cold run time: 98.04 s
Total hot run time: 28.66 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature] doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3859429931
# BE UT Coverage Report
Increment line coverage `13.33% (2/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9f3938347c127a752b04e13d9f40db00f84820c1_9f3938347c127a752b04e13d9f40db00f84820c1/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9f3938347c127a752b04e13d9f40db00f84820c1_9f3938347c127a752b04e13d9f40db00f84820c1/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.61% (19409/36895) |
| Line Coverage | 36.12% (180587/499982) |
| Region Coverage | 32.47% (140040/431269) |
| Branch Coverage | 33.48% (60631/181101) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2780494246
##########
regression-test/suites/external_table_p0/remote_doris/test_insert_remote_doris_as_olap_table_select.groovy:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_insert_remote_doris_as_olap_table_select", "p0,external,doris,external_docker,external_docker_doris") {
+ String remote_doris_host = context.config.otherConfigs.get("extArrowFlightSqlHost")
+ String remote_doris_arrow_port = context.config.otherConfigs.get("extArrowFlightSqlPort")
+ String remote_doris_http_port = context.config.otherConfigs.get("extArrowFlightHttpPort")
+ String remote_doris_user = context.config.otherConfigs.get("extArrowFlightSqlUser")
+ String remote_doris_psw = context.config.otherConfigs.get("extArrowFlightSqlPassword")
+ String remote_doris_thrift_port = context.config.otherConfigs.get("extFeThriftPort")
+
+ def showres = sql "show frontends";
+ remote_doris_arrow_port = showres[0][6]
+ remote_doris_http_port = showres[0][3]
+ remote_doris_thrift_port = showres[0][5]
+ log.info("show frontends log = ${showres}, arrow: ${remote_doris_arrow_port}, http: ${remote_doris_http_port}, thrift: ${remote_doris_thrift_port}")
+
+ def showres2 = sql "show backends";
+ log.info("show backends log = ${showres2}")
+
+ def db_name = "test_insert_remote_doris_as_olap_table_select_db"
+ def catalog_name = "test_insert_remote_doris_as_olap_table_select_catalog"
+
+ sql """DROP DATABASE IF EXISTS ${db_name}"""
+
+ sql """CREATE DATABASE IF NOT EXISTS ${db_name}"""
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ DROP CATALOG IF EXISTS `${catalog_name}`
+ """
+
+
+ sql """
+ CREATE CATALOG `${catalog_name}` PROPERTIES (
+ 'type' = 'doris',
+ 'fe_http_hosts' = 'http://${remote_doris_host}:${remote_doris_http_port}',
+ 'fe_arrow_hosts' = '${remote_doris_host}:${remote_doris_arrow_port}',
+ 'fe_thrift_hosts' = '${remote_doris_host}:${remote_doris_thrift_port}',
+ 'user' = '${remote_doris_user}',
+ 'password' = '${remote_doris_psw}',
+ 'use_arrow_flight' = 'false'
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 03:00:00.000', false, 127, 32767, 2147483647, 9223372036854775807, 1234567890123456789, 123.456, 123456.789, 123457, 123456789012346, 1234567890123456789012345678, '9999-12-31', '9999-12-31 23:59:59', '', '', '', [], {}, {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 04:00:00.000', true, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '2023-10-01', '2023-10-01 12:34:56', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"});
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t2` (
+ `id` datetime(3) NOT NULL,
+ `a_boolean` array<boolean> NULL,
+ `a_tinyint` array<tinyint> NULL,
+ `a_smallint` array<smallint> NULL,
+ `a_int` array<int> NULL,
+ `a_bigint` array<bigint> NULL,
+ `a_largeint` array<largeint> NULL,
+ `a_float` array<float> NULL,
+ `a_double` array<double> NULL,
+ `a_decimal9` array<decimal(9,0)> NULL,
+ `a_decimal18` array<decimal(18,0)> NULL,
+ `a_decimal32` array<decimal(32,0)> NULL,
+ `a_date` array<date> NULL,
+ `a_datetime` array<datetime> NULL,
+ `a_char` array<char(1)> NULL,
+ `a_varchar` array<varchar(65533)> NULL,
+ `a_string` array<text> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648], [-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789], [-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'], ['0000-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!'])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 02:00:00.000', [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 03:00:00.000', [false], [127], [32767], [2147483647], [9223372036854775807], [1234567890123456789], [123.456], [123456.789], [123457], [123456789012346], [1234567890123456789012345678], ['9999-12-31'], ['9999-12-31 23:59:59'], [''], [''], [''])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 04:00:00.000', [true], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], ['2023-10-01'], ['2023-10-01 12:34:56'], ['A'], ['Hello'], ['Hello, Doris!']);
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t3` (
+ `id` datetime NOT NULL,
+ `datetime_0` datetime(0) NULL,
+ `datetime_1` datetime(1) NULL,
+ `datetime_3` datetime(2) NULL,
+ `datetime_4` datetime(3) NULL,
+ `datetime_5` datetime(4) NULL,
+ `datetime_6` datetime(5) NULL,
+ `datetime_7` datetime(6) NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111');
+ """
+
+
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test select after insert also get correct data, it seems we can newly partition version
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111');
+ """
+ qt_query_after_insert """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test insert command
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_insert` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+ // test insert into
+ explain {
+ sql("insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
+ """
+ qt_after_insert_cmd """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` order by id
+ """
+ // test insert overwrite
+ explain {
+ sql("insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
Review Comment:
And why we set these 2 variables?
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -1662,13 +1662,178 @@ struct TGetOlapTableMetaRequest {
5: required i64 table_id
6: optional i32 version // todo serialize according to the version
7: optional list<TPartitionMeta> partitions // client owned partition meta
+ 8: optional list<TPartitionMeta> temp_partitions // client owned partition meta
}
struct TGetOlapTableMetaResult {
1: required Status.TStatus status
2: required binary table_meta
3: optional list<binary> updated_partitions
4: optional list<i64> removed_partitions
+ 5: optional list<binary> updated_temp_partitions
+ 6: optional list<i64> removed_temp_partitions
+}
+
+// Remote transaction request and Result definitions for cross-cluster export.
+// These structs are used by beginRemoteTxn/commitRemoteTxn/abortRemoteTxn RPCs.
+struct TBeginRemoteTxnRequest {
+ 1: required string user
Review Comment:
always use `optional` for all fields
##########
regression-test/suites/external_table_p0/remote_doris/test_insert_remote_doris_as_olap_table_select.groovy:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_insert_remote_doris_as_olap_table_select", "p0,external,doris,external_docker,external_docker_doris") {
+ String remote_doris_host = context.config.otherConfigs.get("extArrowFlightSqlHost")
+ String remote_doris_arrow_port = context.config.otherConfigs.get("extArrowFlightSqlPort")
+ String remote_doris_http_port = context.config.otherConfigs.get("extArrowFlightHttpPort")
+ String remote_doris_user = context.config.otherConfigs.get("extArrowFlightSqlUser")
+ String remote_doris_psw = context.config.otherConfigs.get("extArrowFlightSqlPassword")
+ String remote_doris_thrift_port = context.config.otherConfigs.get("extFeThriftPort")
+
+ def showres = sql "show frontends";
+ remote_doris_arrow_port = showres[0][6]
+ remote_doris_http_port = showres[0][3]
+ remote_doris_thrift_port = showres[0][5]
+ log.info("show frontends log = ${showres}, arrow: ${remote_doris_arrow_port}, http: ${remote_doris_http_port}, thrift: ${remote_doris_thrift_port}")
+
+ def showres2 = sql "show backends";
+ log.info("show backends log = ${showres2}")
+
+ def db_name = "test_insert_remote_doris_as_olap_table_select_db"
+ def catalog_name = "test_insert_remote_doris_as_olap_table_select_catalog"
+
+ sql """DROP DATABASE IF EXISTS ${db_name}"""
+
+ sql """CREATE DATABASE IF NOT EXISTS ${db_name}"""
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ DROP CATALOG IF EXISTS `${catalog_name}`
+ """
+
+
+ sql """
+ CREATE CATALOG `${catalog_name}` PROPERTIES (
+ 'type' = 'doris',
+ 'fe_http_hosts' = 'http://${remote_doris_host}:${remote_doris_http_port}',
+ 'fe_arrow_hosts' = '${remote_doris_host}:${remote_doris_arrow_port}',
+ 'fe_thrift_hosts' = '${remote_doris_host}:${remote_doris_thrift_port}',
+ 'user' = '${remote_doris_user}',
+ 'password' = '${remote_doris_psw}',
+ 'use_arrow_flight' = 'false'
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648, -9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457, -123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01 00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 03:00:00.000', false, 127, 32767, 2147483647, 9223372036854775807, 1234567890123456789, 123.456, 123456.789, 123457, 123456789012346, 1234567890123456789012345678, '9999-12-31', '9999-12-31 23:59:59', '', '', '', [], {}, {11, 3.14, "Emily"})
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` values('2025-05-18 04:00:00.000', true, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '2023-10-01', '2023-10-01 12:34:56', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]', {"Emily":101,"age":25} , {11, 3.14, "Emily"});
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t2` (
+ `id` datetime(3) NOT NULL,
+ `a_boolean` array<boolean> NULL,
+ `a_tinyint` array<tinyint> NULL,
+ `a_smallint` array<smallint> NULL,
+ `a_int` array<int> NULL,
+ `a_bigint` array<bigint> NULL,
+ `a_largeint` array<largeint> NULL,
+ `a_float` array<float> NULL,
+ `a_double` array<double> NULL,
+ `a_decimal9` array<decimal(9,0)> NULL,
+ `a_decimal18` array<decimal(18,0)> NULL,
+ `a_decimal32` array<decimal(32,0)> NULL,
+ `a_date` array<date> NULL,
+ `a_datetime` array<datetime> NULL,
+ `a_char` array<char(1)> NULL,
+ `a_varchar` array<varchar(65533)> NULL,
+ `a_string` array<text> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648], [-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789], [-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'], ['0000-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!'])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 02:00:00.000', [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 03:00:00.000', [false], [127], [32767], [2147483647], [9223372036854775807], [1234567890123456789], [123.456], [123456.789], [123457], [123456789012346], [1234567890123456789012345678], ['9999-12-31'], ['9999-12-31 23:59:59'], [''], [''], [''])
+ """
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` values('2025-05-18 04:00:00.000', [true], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], ['2023-10-01'], ['2023-10-01 12:34:56'], ['A'], ['Hello'], ['Hello, Doris!']);
+ """
+
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_select_t3` (
+ `id` datetime NOT NULL,
+ `datetime_0` datetime(0) NULL,
+ `datetime_1` datetime(1) NULL,
+ `datetime_3` datetime(2) NULL,
+ `datetime_4` datetime(3) NULL,
+ `datetime_5` datetime(4) NULL,
+ `datetime_6` datetime(5) NULL,
+ `datetime_7` datetime(6) NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111', '2025-05-18 01:00:00.111111');
+ """
+
+
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t2` order by id
+ """
+
+ qt_sql """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test select after insert also get correct data, it seems we can newly partition version
+ sql """
+ INSERT INTO `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` values('2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111', '2025-05-19 01:00:00.111111');
+ """
+ qt_query_after_insert """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t3` order by id
+ """
+
+ // test insert command
+ sql """
+ CREATE TABLE `${db_name}`.`test_remote_doris_all_types_insert` (
+ `id` datetime(3) NOT NULL,
+ `c_boolean` boolean NULL,
+ `c_tinyint` tinyint NULL,
+ `c_smallint` smallint NULL,
+ `c_int` int NULL,
+ `c_bigint` bigint NULL,
+ `c_largeint` largeint NULL,
+ `c_float` float NULL,
+ `c_double` double NULL,
+ `c_decimal9` decimal(9,0) NULL,
+ `c_decimal18` decimal(18,0) NULL,
+ `c_decimal32` decimal(32,0) NULL,
+ `c_date` date NULL,
+ `c_datetime` datetime NULL,
+ `c_char` char(1) NULL,
+ `c_varchar` varchar(65533) NULL,
+ `c_string` text NULL,
+ `c_array_s` array<text> NULL,
+ `c_map` MAP<STRING, INT> NULL,
+ `c_struct` STRUCT<f1:INT,f2:FLOAT,f3:STRING> NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+ // test insert into
+ explain {
+ sql("insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert into `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
+ """
+ qt_after_insert_cmd """
+ select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` order by id
+ """
+ // test insert overwrite
+ explain {
+ sql("insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`")
+ contains("VOlapScanNode")
+ }
+ sql """
+ insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
Review Comment:
What is the behavior of setting these variables?
Does it affect remote cluster?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java:
##########
@@ -173,42 +301,462 @@ public RemoteOlapTable getOlapTable(String dbName, String table, long tableId, L
removedPartitions = new ArrayList<>();
}
remoteOlapTable.rebuildPartitions(partitions, updatedPartitions, removedPartitions);
+ // rebuild temp partitions
+ if (result.isSetUpdatedTempPartitions() && result.getUpdatedTempPartitionsSize() > 0) {
+ updatedPartitions = new ArrayList<>(result.getUpdatedTempPartitionsSize());
+ for (ByteBuffer buffer : result.getUpdatedTempPartitions()) {
+ try (ByteArrayInputStream in =
+ new ByteArrayInputStream(buffer.array(), buffer.position(), buffer.remaining());
+ DataInputStream dataInputStream = new DataInputStream(in)) {
+ String partitionStr = Text.readString(dataInputStream);
+ Partition partition = GsonUtils.GSON.fromJson(partitionStr, Partition.class);
+ updatedPartitions.add(partition);
+ }
+ }
+ }
+ removedPartitions = result.getRemovedTempPartitions();
+ if (removedPartitions == null) {
+ removedPartitions = new ArrayList<>();
+ }
+ remoteOlapTable.rebuildTempPartitions(tempPartitions, updatedPartitions, removedPartitions);
return remoteOlapTable;
- }, msg, timeout);
+ }, msg, timeoutMs);
+ }
+
+ public TBeginRemoteTxnResult beginRemoteTxn(TBeginRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to begin remote txn from remote doris:%s, label=%s", name,
+ request.getLabel());
+ long startTime = System.currentTimeMillis();
+ TBeginRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.beginRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("begin remote txn for catalog {} failed, cost={}ms", name, costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("begin remote txn for catalog {} finished, cost={}ms, statusCode={}",
+ name, costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TCommitRemoteTxnResult commitRemoteTxn(TCommitRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to commit remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TCommitRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.commitRemoteTxn(request), msg,
+ Math.max(timeoutMs, (int) request.getInsertVisibleTimeoutMs()));
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("commit remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("commit remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
+ }
+
+ public TAbortRemoteTxnResult abortRemoteTxn(TAbortRemoteTxnRequest request) throws Exception {
+ request.setUser(user);
+ request.setPasswd(password);
+ String msg = String.format("failed to abort remote txn from remote doris:%s, txnId=%d", name,
+ request.getTxnId());
+ long startTime = System.currentTimeMillis();
+ TAbortRemoteTxnResult result;
+ try {
+ result = masterCallWithRetry(client -> client.abortRemoteTxn(request), msg, timeoutMs);
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("abort remote txn for catalog {} failed, txnId={}, cost={}ms",
+ name, request.getTxnId(), costMs, e);
+ throw e;
+ }
+
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.info("abort remote txn for catalog {} finished, txnId={}, cost={}ms, statusCode={}",
+ name, request.getTxnId(), costMs, result.getStatus().getStatusCode());
+ return result;
}
private interface ThriftCall<T> {
public T call(FrontendService.Client client) throws Exception;
}
- private static class MasterResult<T> {
- boolean isMaster = true;
- T result;
- boolean hasError = false;
- String errorMsg;
+ public TNetworkAddress getMasterAddress() throws RuntimeException {
+ TMasterAddressRequest request = new TMasterAddressRequest();
+ request.setUser(user);
+ request.setPasswd(password);
+ long startTime = System.currentTimeMillis();
+ TMasterAddressResult result;
+ try {
+ result = masterCallWithRetry(client -> client.getMasterAddress(request),
+ "failed to get master address from remote doris:" + name, timeoutMs);
+ if (result.isSetMasterAddress()) {
+ master = result.getMasterAddress();
+ }
+ if (result.getStatus().getStatusCode() != TStatusCode.OK) {
+ throw new RuntimeException(result.getStatus().getErrorMsgs().get(0));
+ }
+ } catch (Exception e) {
+ long costMs = System.currentTimeMillis() - startTime;
+ LOG.warn("get master address for catalog {} failed, cost={}ms", name, costMs, e);
+ throw new RuntimeException(e.getMessage());
Review Comment:
```suggestion
throw new RuntimeException(Util.getRootCause(e), e);
```
Same suggestion for other places
##########
gensrc/thrift/FrontendService.thrift:
##########
@@ -1662,13 +1662,178 @@ struct TGetOlapTableMetaRequest {
5: required i64 table_id
6: optional i32 version // todo serialize according to the version
7: optional list<TPartitionMeta> partitions // client owned partition meta
+ 8: optional list<TPartitionMeta> temp_partitions // client owned partition meta
}
struct TGetOlapTableMetaResult {
1: required Status.TStatus status
2: required binary table_meta
3: optional list<binary> updated_partitions
4: optional list<i64> removed_partitions
+ 5: optional list<binary> updated_temp_partitions
+ 6: optional list<i64> removed_temp_partitions
+}
+
+// Remote transaction request and Result definitions for cross-cluster export.
+// These structs are used by beginRemoteTxn/commitRemoteTxn/abortRemoteTxn RPCs.
+struct TBeginRemoteTxnRequest {
+ 1: required string user
+ 2: required string passwd
+ 3: optional string token
+ 4: required string db
Review Comment:
better add `catalog`, even if currently only support "internal" catalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3875385990
# BE Regression && UT Coverage Report
Increment line coverage `13.33% (2/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ef0c94e649ed972209a13ed67e7e65fa49ea4dc9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 57.37% (20733/36139) |
| Line Coverage | 40.29% (200931/498675) |
| Region Coverage | 36.96% (160970/435562) |
| Branch Coverage | 37.71% (68547/181761) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "TsukiokaKogane (via GitHub)" <gi...@apache.org>.
TsukiokaKogane commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2804867361
##########
be/src/exec/tablet_info.cpp:
##########
@@ -447,6 +447,10 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_overwrite_group_id = t_param.overwrite_group_id;
}
+ if (t_param.__isset.master_address) {
+ _master_address = new TNetworkAddress(t_param.master_address);
Review Comment:
fixed
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java:
##########
@@ -56,25 +81,26 @@ public class FeServiceClient {
private final String user;
private final String password;
private final int retryCount;
- private final int timeout;
+ private final int timeoutMs;
+ private volatile TNetworkAddress master;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #60557:
URL: https://github.com/apache/doris/pull/60557#discussion_r2802455882
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java:
##########
@@ -56,25 +81,26 @@ public class FeServiceClient {
private final String user;
private final String password;
private final int retryCount;
- private final int timeout;
+ private final int timeoutMs;
+ private volatile TNetworkAddress master;
Review Comment:
How about using `AtomicReference<TNetworkAddress>`？
##########
be/src/exec/tablet_info.cpp:
##########
@@ -447,6 +447,10 @@ VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSche
_overwrite_group_id = t_param.overwrite_group_id;
}
+ if (t_param.__isset.master_address) {
+ _master_address = new TNetworkAddress(t_param.master_address);
Review Comment:
better use smart pointer instead of raw pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3899532895
# FE Regression Coverage Report
Increment line coverage ` 29.63% (371/1252)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_ee6a26df6cc55709ed3e058d2294a831dc77e153_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3958141589
<details>
<summary>TPC-DS: <b>Total hot run time: 183639 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d552c156312988d13a7e1dc37a84672169ae2f57, data reload: false
query5 4411 654 536 536
query6 463 223 222 222
query7 4224 479 278 278
query8 369 252 245 245
query9 8762 2747 2721 2721
query10 506 406 339 339
query11 16977 17491 17264 17264
query12 188 129 131 129
query13 1297 488 362 362
query14 6838 3418 3067 3067
query14_1 2939 3011 2940 2940
query15 206 200 204 200
query16 991 500 490 490
query17 1422 780 648 648
query18 2983 467 367 367
query19 230 234 200 200
query20 149 142 143 142
query21 238 163 144 144
query22 5568 5012 4938 4938
query23 17315 16729 16568 16568
query23_1 16780 16696 16725 16696
query24 7200 1640 1228 1228
query24_1 1248 1248 1240 1240
query25 563 480 431 431
query26 1233 278 152 152
query27 2732 508 294 294
query28 4399 1864 1838 1838
query29 779 542 455 455
query30 313 244 207 207
query31 867 736 646 646
query32 83 71 70 70
query33 514 337 270 270
query34 919 928 570 570
query35 632 672 576 576
query36 1095 1178 943 943
query37 128 93 86 86
query38 2952 2874 2915 2874
query39 891 877 838 838
query39_1 829 845 818 818
query40 229 153 133 133
query41 60 57 56 56
query42 110 105 101 101
query43 367 387 347 347
query44
query45 194 189 185 185
query46 871 969 593 593
query47 2140 2119 2048 2048
query48 314 307 229 229
query49 624 464 384 384
query50 675 283 211 211
query51 4128 4058 4015 4015
query52 106 107 101 101
query53 291 335 283 283
query54 286 258 253 253
query55 96 89 80 80
query56 302 313 290 290
query57 1398 1354 1294 1294
query58 288 275 289 275
query59 2583 2691 2554 2554
query60 366 328 319 319
query61 142 148 159 148
query62 627 595 529 529
query63 305 277 275 275
query64 4840 1254 965 965
query65
query66 1410 461 343 343
query67 16538 16460 16249 16249
query68
query69 392 319 275 275
query70 976 964 959 959
query71 324 299 297 297
query72 2737 2626 2418 2418
query73 542 540 318 318
query74 9950 9885 9770 9770
query75 2823 2766 2460 2460
query76 2304 1045 684 684
query77 348 381 300 300
query78 11245 11479 10698 10698
query79 1124 781 592 592
query80 1347 625 527 527
query81 553 277 245 245
query82 1339 158 114 114
query83 329 255 241 241
query84 247 119 92 92
query85 918 470 431 431
query86 413 304 287 287
query87 3165 3177 2967 2967
query88 3498 2641 2623 2623
query89 428 369 344 344
query90 1988 171 171 171
query91 163 157 132 132
query92 76 75 75 75
query93 906 830 507 507
query94 631 304 279 279
query95 583 332 313 313
query96 632 524 227 227
query97 2477 2511 2427 2427
query98 224 214 223 214
query99 999 1023 950 950
Total cold run time: 254606 ms
Total hot run time: 183639 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](doris-catalog) doris cross-cluster insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60557:
URL: https://github.com/apache/doris/pull/60557#issuecomment-3959300742
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (16/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60557_d552c156312988d13a7e1dc37a84672169ae2f57_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60557_d552c156312988d13a7e1dc37a84672169ae2f57_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.55% (26133/36525) |
| Line Coverage | 54.30% (273694/503995) |
| Region Coverage | 51.59% (227341/440710) |
| Branch Coverage | 53.04% (97812/184398) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org