You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "CalvinKirs (via GitHub)" <gi...@apache.org> on 2024/02/22 03:00:26 UTC
[PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
CalvinKirs opened a new pull request, #31248:
URL: https://github.com/apache/doris/pull/31248
## What happend
```
def is_increment_change = { def cur_job_name ->
def mv_task_infos = sql """select
^^^^^^^^^^^^^^^^^^^^^^^^^^ERROR LINE^^^^^^^^^^^^^^^^^^^^^^^^^^
JobName, Status, RefreshMode, NeedRefreshPartitions, CompletedPartitions, Progress
from tasks("type"="mv") where JobName="${cur_job_name}" order by CreateTime desc"""
assert (mv_task_infos.size() == 2)
def refresh_info = sql """select
JobName, Status, RefreshMode, NeedRefreshPartitions, CompletedPartitions, Progress
from tasks("type"="mv") where JobName="${cur_job_name}" order by CreateTime desc limit 1;"""
assert (refresh_info[0][0] == cur_job_name)
assert (refresh_info[0][1] == "SUCCESS")
assert (refresh_info[0][2] == "PARTIAL")
Exception:
java.sql.SQLException: errCode = 2, detailMessage = (172.16.0.11)[CANCELLED]failed to call frontend service, reason: Internal error processing fetchSchemaTableData
Issue Number: close #xxx
```
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958626882
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 834eb77bdf9cf96ba9dc0e4e19ef1c076be43ee5 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.4 seconds inserted 10000000 Rows, about 694K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958611846
<details>
<summary>TPC-DS: <b>Total hot run time: 175962 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 834eb77bdf9cf96ba9dc0e4e19ef1c076be43ee5, data reload: false
query1 930 358 345 345
query2 6511 1714 1633 1633
query3 6704 207 210 207
query4 23015 21050 21185 21050
query5 4279 372 375 372
query6 256 164 164 164
query7 4624 303 301 301
query8 257 197 196 196
query9 8467 2816 2772 2772
query10 428 233 219 219
query11 15080 14570 14406 14406
query12 144 83 84 83
query13 1698 420 444 420
query14 9188 7710 7558 7558
query15 214 192 185 185
query16 7508 253 247 247
query17 1421 540 533 533
query18 1953 266 263 263
query19 188 147 142 142
query20 84 80 84 80
query21 195 129 124 124
query22 4883 4727 4689 4689
query23 32410 31601 31591 31591
query24 12695 3395 3375 3375
query25 665 368 361 361
query26 1874 152 156 152
query27 3062 314 314 314
query28 6580 1852 1845 1845
query29 1147 614 633 614
query30 285 142 140 140
query31 943 738 763 738
query32 101 62 57 57
query33 724 229 222 222
query34 1089 499 500 499
query35 943 822 823 822
query36 970 894 882 882
query37 176 65 57 57
query38 3279 3200 3180 3180
query39 1378 1336 1336 1336
query40 287 106 105 105
query41 40 34 35 34
query42 111 103 97 97
query43 464 448 444 444
query44 1065 685 697 685
query45 197 186 176 176
query46 1046 757 775 757
query47 1680 1560 1609 1560
query48 408 344 343 343
query49 1217 313 300 300
query50 770 372 377 372
query51 4430 4382 4255 4255
query52 107 96 95 95
query53 401 298 303 298
query54 292 216 227 216
query55 87 79 84 79
query56 228 206 196 196
query57 1087 924 939 924
query58 221 198 195 195
query59 2217 2064 2190 2064
query60 242 213 224 213
query61 86 86 82 82
query62 595 349 365 349
query63 321 301 293 293
query64 6283 3071 3109 3071
query65 3288 3272 3244 3244
query66 1346 326 338 326
query67 14376 14404 14067 14067
query68 5083 549 578 549
query69 519 356 369 356
query70 1266 1196 1223 1196
query71 432 254 250 250
query72 6306 2788 2612 2612
query73 706 312 312 312
query74 6878 6470 6401 6401
query75 3187 2573 2575 2573
query76 3308 1139 1208 1139
query77 360 236 237 236
query78 9361 8830 8697 8697
query79 968 504 523 504
query80 522 363 348 348
query81 446 208 209 208
query82 161 90 88 88
query83 142 122 119 119
query84 223 81 78 78
query85 1026 355 341 341
query86 304 294 319 294
query87 3425 3295 3260 3260
query88 2694 2275 2273 2273
query89 428 362 340 340
query90 1905 167 164 164
query91 156 125 130 125
query92 56 49 47 47
query93 1015 524 501 501
query94 1102 175 176 175
query95 429 337 337 337
query96 573 266 263 263
query97 4438 4273 4269 4269
query98 222 196 195 195
query99 1065 741 698 698
Total cold run time: 267946 ms
Total hot run time: 175962 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958603424
<details>
<summary>TPC-H: <b>Total hot run time: 40978 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 834eb77bdf9cf96ba9dc0e4e19ef1c076be43ee5, data reload: false
------ Round 1 ----------------------------------
q1 17702 4930 4860 4860
q2 2038 142 136 136
q3 10585 1032 981 981
q4 4652 955 967 955
q5 7602 3146 3161 3146
q6 197 133 133 133
q7 1246 770 753 753
q8 9236 2031 2023 2023
q9 7408 6546 6546 6546
q10 8307 2645 2623 2623
q11 408 223 221 221
q12 753 333 329 329
q13 17911 3607 3612 3607
q14 292 257 254 254
q15 587 506 522 506
q16 479 404 405 404
q17 930 859 858 858
q18 7452 6679 6633 6633
q19 1529 1486 1490 1486
q20 550 282 272 272
q21 6714 3916 3958 3916
q22 868 341 336 336
Total cold run time: 107446 ms
Total hot run time: 40978 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4832 4819 4812 4812
q2 289 185 188 185
q3 3583 3583 3569 3569
q4 2508 2516 2525 2516
q5 5757 5738 5710 5710
q6 213 127 129 127
q7 2230 1661 1685 1661
q8 2976 3061 3071 3061
q9 8724 8714 8725 8714
q10 6798 4235 4204 4204
q11 516 384 366 366
q12 765 546 555 546
q13 4347 3455 3366 3366
q14 262 233 245 233
q15 640 502 490 490
q16 490 442 440 440
q17 1653 1584 1601 1584
q18 8336 7761 7706 7706
q19 1621 1627 1627 1627
q20 2094 1837 1857 1837
q21 6515 6138 6086 6086
q22 592 511 513 511
Total cold run time: 65741 ms
Total hot run time: 59351 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958713872
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958581480
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs merged PR #31248:
URL: https://github.com/apache/doris/pull/31248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958570435
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958572044
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Job)TVF Query JOB Concurrent Reading and Writing Causes Exception [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31248:
URL: https://github.com/apache/doris/pull/31248#issuecomment-1958616381
<details>
<summary>ClickBench: <b>Total hot run time: 30.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 834eb77bdf9cf96ba9dc0e4e19ef1c076be43ee5, data reload: false
query1 0.03 0.02 0.02
query2 0.07 0.02 0.02
query3 0.23 0.07 0.07
query4 1.63 0.08 0.09
query5 0.49 0.48 0.48
query6 1.36 0.62 0.61
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.55 0.45 0.46
query10 0.51 0.48 0.51
query11 0.12 0.09 0.09
query12 0.11 0.09 0.10
query13 0.58 0.59 0.60
query14 0.77 0.79 0.77
query15 0.82 0.80 0.79
query16 0.33 0.33 0.33
query17 0.90 0.91 0.90
query18 0.15 0.18 0.18
query19 1.73 1.70 1.71
query20 0.01 0.01 0.01
query21 15.41 0.64 0.62
query22 3.52 4.46 1.50
query23 17.94 1.12 0.97
query24 2.17 0.30 0.47
query25 0.56 0.13 0.06
query26 0.17 0.14 0.15
query27 0.05 0.06 0.05
query28 11.89 0.86 0.80
query29 12.62 3.50 3.44
query30 0.54 0.51 0.46
query31 2.78 0.36 0.36
query32 3.33 0.48 0.49
query33 3.14 3.12 3.17
query34 15.39 4.52 4.49
query35 4.51 4.50 4.50
query36 1.07 0.94 0.95
query37 0.07 0.05 0.05
query38 0.04 0.04 0.03
query39 0.02 0.02 0.02
query40 0.17 0.14 0.14
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.96 s
Total hot run time: 30.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org