You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/02/20 06:55:02 UTC
[PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
w41ter opened a new pull request, #31144:
URL: https://github.com/apache/doris/pull/31144
## Proposed changes
Issue Number: close #xxx
In upload implementation, filesystem::size throws an exception if the specified file, which is removed by the release snapshot task, does not exist.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953631093
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953667494
<details>
<summary>TPC-DS: <b>Total hot run time: 178214 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85bc2abd6adfc4b8c9afe97dc736098e1d7e0f48, data reload: false
query1 921 343 344 343
query2 6528 1816 1701 1701
query3 6696 202 200 200
query4 23001 21071 21190 21071
query5 4262 461 367 367
query6 264 169 181 169
query7 4599 302 295 295
query8 247 194 209 194
query9 8408 2824 2822 2822
query10 407 207 227 207
query11 14930 14577 14520 14520
query12 139 85 82 82
query13 1698 418 442 418
query14 9024 7591 7521 7521
query15 213 183 192 183
query16 7274 256 253 253
query17 1410 550 536 536
query18 1975 269 266 266
query19 200 147 144 144
query20 88 80 85 80
query21 190 129 122 122
query22 4990 5028 5070 5028
query23 32450 31613 31462 31462
query24 12708 3388 3363 3363
query25 661 368 368 368
query26 1865 149 160 149
query27 3056 318 322 318
query28 6642 1840 1841 1840
query29 1169 607 615 607
query30 285 134 154 134
query31 917 735 762 735
query32 95 59 57 57
query33 737 245 250 245
query34 1295 490 490 490
query35 945 835 821 821
query36 963 974 910 910
query37 160 60 61 60
query38 3377 3210 3163 3163
query39 1372 1326 1329 1326
query40 276 106 105 105
query41 41 35 35 35
query42 108 101 100 100
query43 461 434 445 434
query44 1063 700 707 700
query45 205 177 174 174
query46 1038 801 750 750
query47 1711 1674 1584 1584
query48 413 335 345 335
query49 1214 302 294 294
query50 768 385 390 385
query51 5359 5201 5231 5201
query52 112 90 92 90
query53 384 303 300 300
query54 287 216 226 216
query55 85 79 86 79
query56 215 202 198 198
query57 1085 1000 980 980
query58 222 201 198 198
query59 2230 2035 2197 2035
query60 241 209 218 209
query61 94 83 81 81
query62 593 394 378 378
query63 324 280 281 280
query64 6476 3048 3126 3048
query65 3300 3268 3233 3233
query66 1366 332 317 317
query67 14633 14684 14518 14518
query68 5178 553 551 551
query69 518 357 357 357
query70 1274 1202 1216 1202
query71 433 256 250 250
query72 6281 2780 2664 2664
query73 701 308 310 308
query74 6933 6599 6579 6579
query75 3210 2575 2533 2533
query76 3333 1144 1192 1144
query77 362 231 230 230
query78 9561 8883 8919 8883
query79 971 487 506 487
query80 514 345 337 337
query81 436 202 205 202
query82 158 83 84 83
query83 143 129 121 121
query84 225 77 76 76
query85 1024 340 329 329
query86 293 302 296 296
query87 3474 3307 3317 3307
query88 2687 2270 2274 2270
query89 422 349 342 342
query90 1965 161 160 160
query91 157 131 127 127
query92 52 49 53 49
query93 1007 499 513 499
query94 1121 182 185 182
query95 480 8644 367 367
query96 577 271 267 267
query97 4448 4318 4283 4283
query98 222 213 193 193
query99 1118 764 709 709
Total cold run time: 269743 ms
Total hot run time: 178214 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31144:
URL: https://github.com/apache/doris/pull/31144#discussion_r1495329181
##########
be/src/io/fs/s3_file_system.cpp:
##########
@@ -381,11 +381,11 @@ Status S3FileSystem::upload_impl(const Path& local_file, const Path& remote_file
local_file.native(), full_path(key)));
}
- auto file_size = std::filesystem::file_size(local_file);
+ auto size = handle->GetBytesTransferred();
LOG(INFO) << "Upload " << local_file.native() << " to s3, endpoint=" << _s3_conf.endpoint
<< ", bucket=" << _s3_conf.bucket << ", key=" << key
- << ", duration=" << duration.count() << ", capacity=" << file_size
- << ", tp=" << (file_size) / duration.count();
+ << ", duration=" << duration.count() << ", capacity=" << size
Review Comment:
the word "capacity=" seem not accurate
##########
be/src/io/fs/s3_file_system.cpp:
##########
@@ -381,11 +381,11 @@ Status S3FileSystem::upload_impl(const Path& local_file, const Path& remote_file
local_file.native(), full_path(key)));
}
- auto file_size = std::filesystem::file_size(local_file);
+ auto size = handle->GetBytesTransferred();
LOG(INFO) << "Upload " << local_file.native() << " to s3, endpoint=" << _s3_conf.endpoint
<< ", bucket=" << _s3_conf.bucket << ", key=" << key
- << ", duration=" << duration.count() << ", capacity=" << file_size
- << ", tp=" << (file_size) / duration.count();
+ << ", duration=" << duration.count() << ", capacity=" << size
Review Comment:
the word "capacity=" seems not accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953619946
TeamCity be ut coverage result:
Function Coverage: 35.76% (8556/23927)
Line Coverage: 27.71% (69399/250410)
Region Coverage: 26.83% (36005/134217)
Branch Coverage: 23.65% (18420/77898)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c332e9fd9ed883a7db228c2daf8489c31864550_3c332e9fd9ed883a7db228c2daf8489c31864550/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953620724
<details>
<summary>TPC-H: <b>Total hot run time: 41281 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c332e9fd9ed883a7db228c2daf8489c31864550, data reload: false
------ Round 1 ----------------------------------
q1 17635 4933 4856 4856
q2 2035 144 131 131
q3 10582 1042 997 997
q4 4661 962 986 962
q5 7706 3237 3239 3237
q6 199 135 129 129
q7 1247 786 791 786
q8 9353 2067 2047 2047
q9 7551 6700 6632 6632
q10 8296 2647 2649 2647
q11 419 216 226 216
q12 750 327 327 327
q13 18004 3681 3642 3642
q14 297 252 259 252
q15 614 535 515 515
q16 470 405 423 405
q17 915 846 834 834
q18 7442 6612 6631 6612
q19 1515 1480 1485 1480
q20 622 349 361 349
q21 6500 3886 3935 3886
q22 868 348 339 339
Total cold run time: 107681 ms
Total hot run time: 41281 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4845 4800 4788 4788
q2 290 180 181 180
q3 3589 3564 3577 3564
q4 2501 2494 2498 2494
q5 5781 5738 5791 5738
q6 212 126 124 124
q7 2217 1618 1640 1618
q8 2965 3066 3070 3066
q9 8682 8664 8641 8641
q10 6774 4213 4221 4213
q11 509 361 377 361
q12 766 543 540 540
q13 4197 3430 3391 3391
q14 270 227 230 227
q15 609 526 504 504
q16 495 451 423 423
q17 1654 1584 1588 1584
q18 8321 7710 7680 7680
q19 1632 1630 1617 1617
q20 2111 1827 1824 1824
q21 6502 6116 6114 6114
q22 558 513 511 511
Total cold run time: 65480 ms
Total hot run time: 59202 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953804956
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953675041
<details>
<summary>ClickBench: <b>Total hot run time: 30.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85bc2abd6adfc4b8c9afe97dc736098e1d7e0f48, data reload: false
query1 0.03 0.02 0.03
query2 0.07 0.03 0.03
query3 0.22 0.08 0.08
query4 1.67 0.07 0.09
query5 0.48 0.48 0.48
query6 1.39 0.60 0.60
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.51 0.44 0.45
query10 0.49 0.49 0.50
query11 0.12 0.10 0.09
query12 0.11 0.11 0.10
query13 0.59 0.59 0.57
query14 0.77 0.79 0.78
query15 0.82 0.81 0.80
query16 0.32 0.33 0.33
query17 0.90 0.90 0.90
query18 0.19 0.15 0.16
query19 1.76 1.63 1.64
query20 0.02 0.01 0.01
query21 15.40 0.66 0.59
query22 2.89 3.89 2.11
query23 17.35 1.16 1.03
query24 2.08 0.56 0.31
query25 0.63 0.07 0.06
query26 0.16 0.15 0.14
query27 0.06 0.05 0.05
query28 12.01 0.79 0.84
query29 12.50 3.33 3.27
query30 0.53 0.49 0.49
query31 2.77 0.35 0.37
query32 3.34 0.47 0.48
query33 3.09 3.17 3.17
query34 15.36 4.50 4.52
query35 4.50 4.50 4.50
query36 1.08 0.94 0.93
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.14 0.15
query41 0.07 0.02 0.01
query42 0.02 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.67 s
Total hot run time: 30.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953679382
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 85bc2abd6adfc4b8c9afe97dc736098e1d7e0f48 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953588731
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953588784
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953653185
<details>
<summary>TPC-H: <b>Total hot run time: 41854 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85bc2abd6adfc4b8c9afe97dc736098e1d7e0f48, data reload: false
------ Round 1 ----------------------------------
q1 17705 5451 4904 4904
q2 2057 142 135 135
q3 10706 1034 1038 1034
q4 4721 971 984 971
q5 7721 3256 3284 3256
q6 197 133 125 125
q7 1249 787 788 787
q8 9363 2095 2078 2078
q9 7706 6754 6704 6704
q10 8386 2665 2651 2651
q11 411 213 217 213
q12 796 335 333 333
q13 18023 3694 3671 3671
q14 297 263 261 261
q15 627 511 495 495
q16 549 428 411 411
q17 937 878 863 863
q18 7308 6845 6776 6776
q19 1520 1499 1511 1499
q20 577 341 348 341
q21 6067 3995 4011 3995
q22 419 359 351 351
Total cold run time: 107342 ms
Total hot run time: 41854 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4869 4947 4854 4854
q2 301 186 185 185
q3 3602 3597 3588 3588
q4 2540 2556 2547 2547
q5 5804 5828 5800 5800
q6 212 128 122 122
q7 2252 1671 1664 1664
q8 3004 3124 3125 3124
q9 8737 8738 8710 8710
q10 4168 4254 4250 4250
q11 528 374 372 372
q12 780 550 548 548
q13 4177 3416 3423 3416
q14 258 242 232 232
q15 630 512 509 509
q16 502 460 437 437
q17 1702 1626 1619 1619
q18 8343 7760 7659 7659
q19 1632 1653 1645 1645
q20 2121 1862 1834 1834
q21 6587 6220 6172 6172
q22 583 537 524 524
Total cold run time: 63332 ms
Total hot run time: 59811 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953598350
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953620423
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953648897
TeamCity be ut coverage result:
Function Coverage: 35.76% (8556/23927)
Line Coverage: 27.71% (69396/250423)
Region Coverage: 26.82% (36004/134225)
Branch Coverage: 23.64% (18419/77908)
Coverage Report: http://coverage.selectdb-in.cc/coverage/85bc2abd6adfc4b8c9afe97dc736098e1d7e0f48_85bc2abd6adfc4b8c9afe97dc736098e1d7e0f48/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #31144:
URL: https://github.com/apache/doris/pull/31144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31144:
URL: https://github.com/apache/doris/pull/31144#discussion_r1495329181
##########
be/src/io/fs/s3_file_system.cpp:
##########
@@ -381,11 +381,11 @@ Status S3FileSystem::upload_impl(const Path& local_file, const Path& remote_file
local_file.native(), full_path(key)));
}
- auto file_size = std::filesystem::file_size(local_file);
+ auto size = handle->GetBytesTransferred();
LOG(INFO) << "Upload " << local_file.native() << " to s3, endpoint=" << _s3_conf.endpoint
<< ", bucket=" << _s3_conf.bucket << ", key=" << key
- << ", duration=" << duration.count() << ", capacity=" << file_size
- << ", tp=" << (file_size) / duration.count();
+ << ", duration=" << duration.count() << ", capacity=" << size
Review Comment:
the word "capacity=" seems not accurate
also, consider change "tp=" to a more understandable name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](backup) fix concurrent upload and release snapshot crash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31144:
URL: https://github.com/apache/doris/pull/31144#issuecomment-1953804892
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org