You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "whutpencil (via GitHub)" <gi...@apache.org> on 2024/01/09 11:49:02 UTC
[PR] [fix](executor) Fe publish topic info tcp leak [doris]
whutpencil opened a new pull request, #29739:
URL: https://github.com/apache/doris/pull/29739
## Proposed changes
Issue Number: close #xxx
After publishing topic info to be, FE did not return the thrust client correctly, resulting in a TCP connection leak.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1882934132
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1882963482
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1882932949
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on code in PR #29739:
URL: https://github.com/apache/doris/pull/29739#discussion_r1446005128
##########
fe/fe-core/src/main/java/org/apache/doris/common/publish/TopicPublisherThread.java:
##########
@@ -103,16 +103,25 @@ public TopicPublishWorker(TPublishTopicRequest request, Backend node, ResponseHa
@Override
public void run() {
long beginTime = System.currentTimeMillis();
+ BackendService.Client client = null;
+ TNetworkAddress address = null;
+ boolean ok = false;
try {
- TNetworkAddress addr = new TNetworkAddress(be.getHost(), be.getBePort());
- BackendService.Client client = ClientPool.backendPool.borrowObject(addr);
+ address = new TNetworkAddress(be.getHost(), be.getBePort());
+ client = ClientPool.backendPool.borrowObject(address);
client.publishTopicInfo(request);
+ ok = true;
LOG.info("publish topic info to be {} success, time cost={} ms",
be.getHost(), (System.currentTimeMillis() - beginTime));
} catch (Exception e) {
LOG.warn("publish topic info to be {} error happens: , time cost={} ms",
be.getHost(), (System.currentTimeMillis() - beginTime), e);
} finally {
+ if (ok) {
+ ClientPool.backendPool.returnObject(address, client);
Review Comment:
We'd better catch a throwable exception for returnObject/invalidateObject
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1883013333
<details>
<summary>ClickBench: <b>Total hot run time: 31.58 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a22d958879b6465a9b1a8411f8ae503ae4d3cd0f, data reload: false
query1 0.07 0.06 0.06
query2 0.06 0.03 0.03
query3 0.32 0.24 0.23
query4 1.72 0.21 0.20
query5 0.53 0.52 0.52
query6 1.55 0.68 0.68
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.59 0.57 0.57
query10 0.61 0.63 0.61
query11 0.18 0.12 0.12
query12 0.16 0.12 0.12
query13 0.60 0.61 0.60
query14 0.76 0.80 0.78
query15 0.81 0.81 0.80
query16 0.35 0.36 0.35
query17 1.00 0.99 1.00
query18 0.25 0.25 0.25
query19 1.89 1.73 1.78
query20 0.01 0.01 0.02
query21 15.42 0.55 0.52
query22 2.94 3.14 1.25
query23 17.38 1.19 1.04
query24 10.01 0.80 0.32
query25 1.81 0.17 0.17
query26 0.14 0.17 0.16
query27 0.17 0.17 0.18
query28 5.75 0.92 0.85
query29 12.99 3.64 3.57
query30 0.52 0.52 0.47
query31 2.91 0.37 0.38
query32 3.31 0.50 0.48
query33 3.20 3.23 3.22
query34 16.60 4.45 4.49
query35 4.44 4.60 4.53
query36 1.15 1.08 1.08
query37 0.07 0.05 0.04
query38 0.04 0.02 0.03
query39 0.02 0.02 0.01
query40 0.17 0.13 0.14
query41 0.07 0.01 0.02
query42 0.02 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 110.67 s
Total hot run time: 31.58 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1883009350
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.38 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s
storage size: 17184249609 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1882989650
<details>
<summary>TPC-H: <b>Total hot run time: 39272 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a22d958879b6465a9b1a8411f8ae503ae4d3cd0f, data reload: false
------ Round 1 ----------------------------------
q1 18350 5543 5499 5499
q2 2038 158 147 147
q3 10663 1133 1203 1133
q4 10245 811 815 811
q5 7814 2932 2868 2868
q6 205 129 135 129
q7 908 513 510 510
q8 9325 2041 2000 2000
q9 6477 6437 6340 6340
q10 8202 3031 2967 2967
q11 438 218 211 211
q12 387 226 228 226
q13 18269 3497 3455 3455
q14 240 213 208 208
q15 552 518 509 509
q16 475 437 414 414
q17 922 736 673 673
q18 7258 6626 6641 6626
q19 1592 1513 1506 1506
q20 712 317 305 305
q21 2825 2403 2402 2402
q22 382 333 340 333
Total cold run time: 108279 ms
Total hot run time: 39272 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5541 5555 5472 5472
q2 351 248 248 248
q3 3365 3305 3326 3305
q4 2121 2070 2067 2067
q5 5842 5868 5852 5852
q6 206 129 124 124
q7 2321 1885 1873 1873
q8 3412 3523 3548 3523
q9 8985 8860 8880 8860
q10 3820 3867 3873 3867
q11 594 508 503 503
q12 831 631 646 631
q13 5843 3234 3252 3234
q14 301 284 265 265
q15 552 513 509 509
q16 537 490 496 490
q17 2372 2375 2365 2365
q18 8661 8622 8374 8374
q19 1554 1606 1613 1606
q20 2201 1959 1945 1945
q21 5642 5368 5412 5368
q22 596 516 524 516
Total cold run time: 65648 ms
Total hot run time: 60997 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #29739:
URL: https://github.com/apache/doris/pull/29739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1882934073
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](executor) Fe publish topic info tcp leak [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29739:
URL: https://github.com/apache/doris/pull/29739#issuecomment-1882964311
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org