You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2024/02/20 09:29:37 UTC
[PR] [Bug](json reader) object should stop processing when encounter error [doris]
eldenmoon opened a new pull request, #31159:
URL: https://github.com/apache/doris/pull/31159
If DATA_QUALITY_ERROR encountered we should stop processing this document any more.Otherwise there will be UB in simdjson.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1955743811
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953896300
<details>
<summary>ClickBench: <b>Total hot run time: 30.72 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e2291868fa9d43973d4c4e4e0b0c21098d62244, data reload: false
query1 0.03 0.02 0.03
query2 0.06 0.03 0.02
query3 0.22 0.08 0.08
query4 1.64 0.08 0.08
query5 0.49 0.49 0.48
query6 1.35 0.61 0.61
query7 0.02 0.01 0.01
query8 0.04 0.03 0.02
query9 0.52 0.43 0.46
query10 0.49 0.47 0.49
query11 0.12 0.09 0.10
query12 0.12 0.11 0.10
query13 0.59 0.59 0.58
query14 0.76 0.78 0.78
query15 0.80 0.80 0.79
query16 0.33 0.33 0.33
query17 0.89 0.91 0.94
query18 0.18 0.16 0.18
query19 1.73 1.67 1.67
query20 0.01 0.01 0.01
query21 15.41 0.63 0.59
query22 3.08 4.49 2.29
query23 17.45 1.08 1.01
query24 2.01 0.94 0.18
query25 0.55 0.10 0.06
query26 0.17 0.14 0.14
query27 0.05 0.05 0.05
query28 11.90 0.86 0.83
query29 12.52 3.37 3.33
query30 0.59 0.49 0.48
query31 2.78 0.35 0.37
query32 3.36 0.47 0.48
query33 3.16 3.12 3.18
query34 15.36 4.51 4.48
query35 4.52 4.49 4.49
query36 1.08 0.94 0.93
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.17 0.16 0.15
query41 0.07 0.01 0.01
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 104.8 s
Total hot run time: 30.72 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953843847
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953866040
<details>
<summary>TPC-H: <b>Total hot run time: 41609 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e2291868fa9d43973d4c4e4e0b0c21098d62244, data reload: false
------ Round 1 ----------------------------------
q1 17635 4939 4861 4861
q2 2048 140 140 140
q3 12151 1046 1046 1046
q4 6004 1046 1042 1042
q5 8665 3243 3276 3243
q6 198 129 130 129
q7 1258 797 775 775
q8 9265 2081 2057 2057
q9 7565 6721 6694 6694
q10 8328 2653 2648 2648
q11 439 219 204 204
q12 742 322 336 322
q13 17981 3652 3667 3652
q14 295 256 257 256
q15 579 508 497 497
q16 479 403 424 403
q17 928 872 880 872
q18 7435 6775 6656 6656
q19 1530 1496 1477 1477
q20 644 355 332 332
q21 6943 3961 3957 3957
q22 882 352 346 346
Total cold run time: 111994 ms
Total hot run time: 41609 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4800 4803 4800 4800
q2 296 184 182 182
q3 3593 3573 3590 3573
q4 2526 2533 2546 2533
q5 5752 5764 5747 5747
q6 211 128 138 128
q7 2230 1623 1649 1623
q8 2992 3062 3070 3062
q9 8683 8707 8696 8696
q10 6936 4234 4239 4234
q11 528 360 373 360
q12 777 548 551 548
q13 4182 3372 3409 3372
q14 261 237 233 233
q15 598 528 498 498
q16 487 427 439 427
q17 1666 1595 1598 1595
q18 8300 7865 7775 7775
q19 1632 1657 1639 1639
q20 2128 1816 1826 1816
q21 6508 6161 6217 6161
q22 578 487 507 487
Total cold run time: 65664 ms
Total hot run time: 59489 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953874816
TeamCity be ut coverage result:
Function Coverage: 35.76% (8557/23927)
Line Coverage: 27.72% (69409/250424)
Region Coverage: 26.83% (36019/134225)
Branch Coverage: 23.64% (18419/77908)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6e2291868fa9d43973d4c4e4e0b0c21098d62244_6e2291868fa9d43973d4c4e4e0b0c21098d62244/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953886428
<details>
<summary>TPC-DS: <b>Total hot run time: 177878 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e2291868fa9d43973d4c4e4e0b0c21098d62244, data reload: false
query1 918 340 350 340
query2 6524 1865 1697 1697
query3 6691 212 206 206
query4 23162 21136 21093 21093
query5 4292 374 372 372
query6 266 169 175 169
query7 4609 295 293 293
query8 251 195 200 195
query9 8452 2805 2789 2789
query10 407 228 222 222
query11 15014 14769 14478 14478
query12 138 86 89 86
query13 1676 411 419 411
query14 8887 7676 7482 7482
query15 212 187 197 187
query16 7105 255 250 250
query17 1415 554 538 538
query18 2096 271 258 258
query19 186 143 142 142
query20 87 85 83 83
query21 189 125 119 119
query22 5201 5102 4988 4988
query23 32613 31450 31576 31450
query24 12816 3418 3408 3408
query25 658 357 361 357
query26 1879 154 162 154
query27 3044 322 323 322
query28 6550 1861 1847 1847
query29 1236 612 603 603
query30 288 137 153 137
query31 924 751 769 751
query32 95 57 60 57
query33 717 238 234 234
query34 1090 486 492 486
query35 939 825 817 817
query36 977 937 915 915
query37 183 62 63 62
query38 3300 3183 3265 3183
query39 1377 1331 1322 1322
query40 287 106 107 106
query41 37 36 34 34
query42 100 100 101 100
query43 475 432 449 432
query44 1077 693 709 693
query45 202 178 173 173
query46 1044 775 757 757
query47 1742 1594 1593 1593
query48 419 342 344 342
query49 1273 307 294 294
query50 762 375 369 369
query51 5286 5140 5182 5140
query52 107 90 91 90
query53 388 303 304 303
query54 292 218 240 218
query55 81 83 76 76
query56 222 204 198 198
query57 1127 957 997 957
query58 209 207 199 199
query59 2338 2261 2153 2153
query60 245 232 226 226
query61 92 83 83 83
query62 601 394 383 383
query63 319 285 286 285
query64 6413 3052 3115 3052
query65 3252 3255 3244 3244
query66 1379 325 322 322
query67 14958 14454 14281 14281
query68 5117 562 541 541
query69 515 351 355 351
query70 1242 1191 1284 1191
query71 366 260 253 253
query72 6325 2766 2630 2630
query73 704 307 303 303
query74 6944 6599 6515 6515
query75 3159 2546 2538 2538
query76 3259 1167 1202 1167
query77 340 233 237 233
query78 9592 8981 8850 8850
query79 956 498 493 493
query80 519 364 346 346
query81 396 206 205 205
query82 162 82 84 82
query83 140 118 118 118
query84 227 78 78 78
query85 763 344 338 338
query86 291 309 306 306
query87 3490 3362 3334 3334
query88 2648 2267 2264 2264
query89 431 349 350 349
query90 2055 165 166 165
query91 151 129 128 128
query92 54 52 48 48
query93 950 522 469 469
query94 1154 193 184 184
query95 8491 380 365 365
query96 563 269 264 264
query97 4402 4267 4255 4255
query98 219 212 189 189
query99 1093 803 720 720
Total cold run time: 277909 ms
Total hot run time: 177878 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953849943
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953901845
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 6e2291868fa9d43973d4c4e4e0b0c21098d62244 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #31159:
URL: https://github.com/apache/doris/pull/31159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953831168
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953806331
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953821834
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](json reader) object should stop processing when encounter error [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31159:
URL: https://github.com/apache/doris/pull/31159#issuecomment-1953839413
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org