You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/11/16 13:54:46 UTC
[PR] [fix](json_reader) fix simd json reader lose data [doris]
HHoflittlefish777 opened a new pull request, #27145:
URL: https://github.com/apache/doris/pull/27145
## Proposed changes
when I do not set read_json_by_line, and I would like to load json, only one json loaded.
But there more than one json, means lose data when load json.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1827040942
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1815966477
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816740400
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.12 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17101511571 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855803633
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855798046
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1859188246
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 48b5d75876ffa64a1bfe6704524216c4c00ebb09, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4681 4446 4472 4446
q2 367 147 157 147
q3 1459 1243 1212 1212
q4 1120 900 896 896
q5 3169 3163 3143 3143
q6 246 128 127 127
q7 989 488 485 485
q8 2233 2236 2199 2199
q9 6680 6660 6645 6645
q10 3208 3263 3305 3263
q11 320 200 203 200
q12 347 213 202 202
q13 4560 3832 3805 3805
q14 242 213 214 213
q15 599 538 520 520
q16 444 384 388 384
q17 1026 665 584 584
q18 7144 6899 6997 6899
q19 1544 1458 1361 1361
q20 560 320 300 300
q21 3035 2655 2639 2639
q22 353 280 289 280
Total cold run time: 44326 ms
Total hot run time: 39950 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4442 4412 4405 4405
q2 266 166 173 166
q3 3547 3531 3525 3525
q4 2389 2376 2370 2370
q5 5733 5740 5720 5720
q6 242 122 123 122
q7 2413 1891 1851 1851
q8 3518 3534 3534 3534
q9 9045 9038 9042 9038
q10 3907 3996 3983 3983
q11 501 387 370 370
q12 770 586 609 586
q13 4370 3575 3543 3543
q14 280 253 264 253
q15 565 520 529 520
q16 487 426 460 426
q17 1892 1868 1860 1860
q18 8713 8184 8403 8184
q19 1785 1747 1751 1747
q20 2257 1953 1925 1925
q21 6526 6215 6137 6137
q22 521 437 419 419
Total cold run time: 64169 ms
Total hot run time: 60684 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1862362440
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1815954928
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1862373038
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #27145:
URL: https://github.com/apache/doris/pull/27145#discussion_r1432175388
##########
regression-test/suites/load_p0/stream_load/test_json_load.groovy:
##########
@@ -794,7 +793,20 @@ suite("test_json_load", "p0") {
assertEquals("${reason}", "${out}")
}
}
+ } finally {
+ try_sql("DROP TABLE IF EXISTS ${testTable}")
+ }
+
+ // iterate read json when read_json_by_line = false
+ try {
+ sql "DROP TABLE IF EXISTS ${testTable}"
Review Comment:
add cases like `iterate_read_json.json` but contains malformed json lines in the middle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1814492013
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1814663220
TeamCity be ut coverage result:
Function Coverage: 36.75% (8410/22883)
Line Coverage: 29.27% (68395/233666)
Region Coverage: 27.86% (35344/126882)
Branch Coverage: 24.62% (18064/73366)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8b468bcb5170c80a8af43123c05a08d02e27b480_8b468bcb5170c80a8af43123c05a08d02e27b480/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #27145:
URL: https://github.com/apache/doris/pull/27145#discussion_r1431394418
##########
be/src/vec/exec/format/json/new_json_reader.cpp:
##########
@@ -1046,19 +1047,25 @@ Status NewJsonReader::_simdjson_handle_simple_json(RuntimeState* /*state*/, Bloc
return Status::OK();
}
- // step2: get json value by json doc
- Status st = _get_json_value(&size, eof, &error, is_empty_row);
- if (st.is<DATA_QUALITY_ERROR>()) {
- return Status::OK();
- }
- RETURN_IF_ERROR(st);
- if (*is_empty_row) {
- return Status::OK();
- }
+ for (_json_stream_iterator = _json_stream.begin();
+ _json_stream_iterator != _json_stream.end(); ++_json_stream_iterator) {
+ if (_json_stream_iterator.current_index() >= _original_doc_size) {
Review Comment:
when will `if (_json_stream_iterator.current_index() >= _original_doc_size)` happened?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1954083523
<details>
<summary>ClickBench: <b>Total hot run time: 31.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6d5c3bc10708aedb7ba5aff316e5f6e38039abb, data reload: false
query1 0.03 0.02 0.02
query2 0.07 0.03 0.02
query3 0.22 0.07 0.07
query4 1.63 0.09 0.09
query5 0.49 0.49 0.49
query6 1.39 0.62 0.62
query7 0.03 0.01 0.01
query8 0.04 0.03 0.03
query9 0.55 0.45 0.46
query10 0.49 0.50 0.49
query11 0.12 0.09 0.10
query12 0.12 0.10 0.10
query13 0.58 0.59 0.58
query14 0.76 0.81 0.78
query15 0.82 0.80 0.78
query16 0.33 0.34 0.34
query17 0.90 0.90 0.92
query18 0.18 0.19 0.19
query19 1.81 1.73 1.70
query20 0.01 0.01 0.02
query21 15.40 0.62 0.60
query22 2.95 4.28 2.64
query23 17.27 1.15 1.01
query24 2.16 0.23 0.55
query25 0.57 0.07 0.06
query26 0.17 0.14 0.14
query27 0.06 0.05 0.06
query28 11.95 0.82 0.84
query29 12.57 3.35 3.31
query30 0.55 0.47 0.48
query31 2.78 0.36 0.36
query32 3.32 0.48 0.48
query33 3.08 3.13 3.14
query34 15.35 4.47 4.47
query35 4.47 4.51 4.47
query36 1.09 0.97 0.98
query37 0.08 0.05 0.05
query38 0.04 0.03 0.02
query39 0.03 0.02 0.01
query40 0.18 0.14 0.14
query41 0.08 0.01 0.01
query42 0.03 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.77 s
Total hot run time: 31.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1954088469
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit b6d5c3bc10708aedb7ba5aff316e5f6e38039abb with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.4 seconds inserted 10000000 Rows, about 694K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855846437
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 2fce1f885a29cf4dca5ccd85aac893577f5f3867, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4710 4463 4508 4463
q2 363 151 159 151
q3 1453 1250 1210 1210
q4 1126 909 937 909
q5 3160 3139 3177 3139
q6 245 130 127 127
q7 999 491 485 485
q8 2226 2196 2225 2196
q9 6707 6631 6632 6631
q10 3225 3263 3267 3263
q11 336 210 212 210
q12 348 208 207 207
q13 4576 3822 3789 3789
q14 240 215 214 214
q15 569 521 517 517
q16 439 395 379 379
q17 1017 565 626 565
q18 7207 6930 6975 6930
q19 1508 1370 1438 1370
q20 546 330 304 304
q21 3079 2609 2608 2608
q22 350 279 284 279
Total cold run time: 44429 ms
Total hot run time: 39946 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4428 4442 4369 4369
q2 267 164 168 164
q3 3538 3545 3526 3526
q4 2388 2378 2365 2365
q5 5727 5737 5760 5737
q6 245 121 122 121
q7 2393 1885 1854 1854
q8 3516 3528 3544 3528
q9 9061 9039 9016 9016
q10 3901 3963 3977 3963
q11 506 393 392 392
q12 774 593 588 588
q13 4264 3582 3565 3565
q14 277 253 254 253
q15 574 515 527 515
q16 503 435 466 435
q17 1869 1861 1894 1861
q18 8701 8239 8340 8239
q19 1703 1728 1734 1728
q20 2249 1936 1937 1936
q21 6557 6158 6204 6158
q22 509 447 438 438
Total cold run time: 63950 ms
Total hot run time: 60751 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855786628
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855804460
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1954075033
<details>
<summary>TPC-DS: <b>Total hot run time: 185290 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6d5c3bc10708aedb7ba5aff316e5f6e38039abb, data reload: false
query1 928 353 353 353
query2 6502 1770 1778 1770
query3 6694 202 198 198
query4 23242 21087 21087 21087
query5 4243 448 455 448
query6 275 167 162 162
query7 4609 288 304 288
query8 258 193 199 193
query9 8468 2850 2852 2850
query10 421 232 247 232
query11 15042 14526 14454 14454
query12 136 84 82 82
query13 1711 440 410 410
query14 9133 7659 7572 7572
query15 213 189 190 189
query16 7184 255 252 252
query17 1414 554 534 534
query18 2058 271 263 263
query19 197 141 146 141
query20 88 84 88 84
query21 193 121 125 121
query22 4923 4732 4730 4730
query23 32445 31513 31436 31436
query24 12973 3359 3412 3359
query25 654 374 363 363
query26 1874 154 163 154
query27 3045 312 314 312
query28 6753 1859 1832 1832
query29 1219 612 610 610
query30 277 137 150 137
query31 944 756 761 756
query32 104 62 54 54
query33 714 244 238 238
query34 1067 478 506 478
query35 933 830 823 823
query36 1014 892 889 889
query37 171 57 60 57
query38 3278 3211 3151 3151
query39 1364 1336 1324 1324
query40 285 113 105 105
query41 38 34 34 34
query42 111 98 101 98
query43 465 436 457 436
query44 1064 697 708 697
query45 197 176 180 176
query46 1023 788 750 750
query47 1652 1535 1628 1535
query48 412 339 356 339
query49 1207 299 294 294
query50 760 387 385 385
query51 5329 5115 5168 5115
query52 107 88 90 88
query53 393 288 291 288
query54 296 219 218 218
query55 83 81 81 81
query56 221 206 193 193
query57 1043 958 960 958
query58 217 204 197 197
query59 2258 2108 2076 2076
query60 261 217 217 217
query61 85 95 84 84
query62 592 383 380 380
query63 323 283 281 281
query64 6536 3074 3128 3074
query65 3267 3238 3232 3232
query66 1376 330 326 326
query67 14678 14580 14364 14364
query68 5135 592 554 554
query69 504 353 357 353
query70 1284 1234 1195 1195
query71 370 254 263 254
query72 6377 2799 2604 2604
query73 711 310 309 309
query74 6964 6394 6505 6394
query75 3204 2548 2566 2548
query76 3086 1145 1186 1145
query77 362 241 227 227
query78 9367 8762 8828 8762
query79 1041 511 492 492
query80 518 347 342 342
query81 446 207 209 207
query82 235 86 90 86
query83 147 119 126 119
query84 227 76 77 76
query85 1054 356 337 337
query86 288 296 313 296
query87 3481 3275 3251 3251
query88 2675 2272 2266 2266
query89 439 350 359 350
query90 1918 162 162 162
query91 157 128 129 128
query92 55 50 47 47
query93 998 486 485 485
query94 1170 188 178 178
query95 8780 8399 8322 8322
query96 582 264 263 263
query97 4421 4248 4301 4248
query98 229 201 193 193
query99 1102 706 690 690
Total cold run time: 278367 ms
Total hot run time: 185290 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1954045738
TeamCity be ut coverage result:
Function Coverage: 35.77% (8558/23927)
Line Coverage: 27.72% (69409/250424)
Region Coverage: 26.83% (36014/134238)
Branch Coverage: 23.65% (18424/77918)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b6d5c3bc10708aedb7ba5aff316e5f6e38039abb_b6d5c3bc10708aedb7ba5aff316e5f6e38039abb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816564693
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1817369895
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 8e32fad60e763d2c0ae7f22c6fef0e314ea63382, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4922 4651 4651 4651
q2 372 142 168 142
q3 2032 1879 1933 1879
q4 1380 1298 1275 1275
q5 3962 3951 4052 3951
q6 253 129 133 129
q7 1439 887 893 887
q8 2793 2789 2770 2770
q9 9725 9432 9607 9432
q10 3475 3495 3525 3495
q11 372 249 238 238
q12 441 292 292 292
q13 4580 3844 3778 3778
q14 326 298 292 292
q15 567 527 528 527
q16 672 597 586 586
q17 1133 969 961 961
q18 7899 7416 7386 7386
q19 1683 1680 1651 1651
q20 543 303 290 290
q21 4450 4018 4036 4018
q22 480 364 367 364
Total cold run time: 53499 ms
Total hot run time: 48994 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4576 4602 4543 4543
q2 337 231 233 231
q3 4038 4042 4017 4017
q4 2715 2714 2703 2703
q5 9578 9599 9562 9562
q6 247 130 125 125
q7 3027 2500 2502 2500
q8 4410 4421 4445 4421
q9 12935 12927 12899 12899
q10 4063 4159 4136 4136
q11 803 643 728 643
q12 970 831 816 816
q13 4309 3595 3528 3528
q14 384 345 346 345
q15 577 523 522 522
q16 732 671 669 669
q17 3960 3829 3859 3829
q18 9536 8990 9152 8990
q19 1787 1760 1785 1760
q20 2402 2059 2047 2047
q21 8953 8581 8726 8581
q22 875 837 774 774
Total cold run time: 81214 ms
Total hot run time: 77641 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855784830
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1857266921
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 2fce1f885a29cf4dca5ccd85aac893577f5f3867, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4779 4440 4463 4440
q2 365 149 147 147
q3 1453 1209 1200 1200
q4 1124 912 866 866
q5 3202 3229 3198 3198
q6 246 127 126 126
q7 1015 490 486 486
q8 2177 2192 2181 2181
q9 6700 6647 6648 6647
q10 3210 3274 3262 3262
q11 320 210 210 210
q12 357 209 208 208
q13 4564 3780 3814 3780
q14 239 211 209 209
q15 568 530 523 523
q16 445 386 390 386
q17 1029 642 573 573
q18 8720 7278 7112 7112
q19 1542 1418 1436 1418
q20 998 314 308 308
q21 3062 2650 2685 2650
q22 348 283 284 283
Total cold run time: 46463 ms
Total hot run time: 40213 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4423 4399 4418 4399
q2 270 163 167 163
q3 3529 3510 3489 3489
q4 2401 2396 2386 2386
q5 5776 5744 5741 5741
q6 235 121 123 121
q7 2363 1867 1831 1831
q8 3513 3516 3526 3516
q9 9066 9007 9051 9007
q10 3923 4014 4012 4012
q11 501 394 373 373
q12 761 576 596 576
q13 4295 3590 3550 3550
q14 287 261 264 261
q15 573 526 529 526
q16 507 496 497 496
q17 1894 1866 1843 1843
q18 8579 7956 8088 7956
q19 1759 1748 1760 1748
q20 2298 1938 1930 1930
q21 6494 6223 6163 6163
q22 494 427 422 422
Total cold run time: 63941 ms
Total hot run time: 60509 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #27145:
URL: https://github.com/apache/doris/pull/27145#discussion_r1432174063
##########
be/src/vec/exec/format/json/new_json_reader.cpp:
##########
@@ -1279,16 +1298,14 @@ Status NewJsonReader::_simdjson_handle_nested_complex_json(
// so that the caller will continue reading next line.
*is_empty_row = true;
}
- break; // read a valid row
- } catch (simdjson::simdjson_error& e) {
- RETURN_IF_ERROR(_handle_simdjson_error(e, block, num_rows, eof));
- if (*_scanner_eof) {
- // When _scanner_eof is true and valid is false, it means that we have encountered
- // unqualified data and decided to stop the scan.
- *is_empty_row = true;
- return Status::OK();
- }
- continue;
+ }
Review Comment:
why remove this break?
##########
be/src/vec/exec/format/json/new_json_reader.cpp:
##########
@@ -1236,20 +1249,26 @@ Status NewJsonReader::_simdjson_handle_nested_complex_json(
RuntimeState* /*state*/, Block& block, const std::vector<SlotDescriptor*>& slot_descs,
bool* is_empty_row, bool* eof) {
// nested complex json
- while (true) {
- size_t num_rows = block.rows();
- simdjson::ondemand::object cur;
- size_t size = 0;
- simdjson::error_code error;
- try {
- RETURN_IF_ERROR(_simdjson_parse_json(&size, is_empty_row, eof, &error));
- if (size == 0 || *eof) {
- *is_empty_row = true;
- return Status::OK();
+ size_t num_rows = block.rows();
+ simdjson::ondemand::object cur;
+ size_t size = 0;
+ simdjson::error_code error;
+
+ try {
+ RETURN_IF_ERROR(_simdjson_parse_json(&size, is_empty_row, eof, &error));
+ if (size == 0 || *eof) {
+ *is_empty_row = true;
+ return Status::OK();
+ }
+
+ for (_json_stream_iterator = _json_stream.begin();
+ _json_stream_iterator != _json_stream.end(); ++_json_stream_iterator) {
+ if (_json_stream_iterator.current_index() >= _original_doc_size) {
+ break;
}
Status st = _get_json_value(&size, eof, &error, is_empty_row);
if (st.is<DATA_QUALITY_ERROR>()) {
- continue; // continue to read next
+ return Status::OK();
Review Comment:
why return OK instead of read next?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1953972306
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1827040917
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855789447
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855799768
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855800661
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1859190718
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1879928981
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 closed pull request #27145: [improve](simd-json-reader) fix simd json reader lose data and support stream parser
URL: https://github.com/apache/doris/pull/27145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816683175
TeamCity be ut coverage result:
Function Coverage: 36.78% (8432/22927)
Line Coverage: 29.28% (68602/234291)
Region Coverage: 27.86% (35459/127264)
Branch Coverage: 24.59% (18094/73580)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6879db1185649e1f28f7933c39301ffd78cdffeb_6879db1185649e1f28f7933c39301ffd78cdffeb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27145:
URL: https://github.com/apache/doris/pull/27145#discussion_r1397406580
##########
be/src/vec/exec/format/json/new_json_reader.h:
##########
@@ -137,7 +137,8 @@ class NewJsonReader : public GenericReader {
static std::string _print_json_value(const rapidjson::Value& value);
- Status _read_one_message(std::unique_ptr<uint8_t[]>* file_buf, size_t* read_size);
+ Status _read_one_message(std::unique_ptr<uint8_t[]>* file_buf, size_t* read_size,
+ size_t* total_length = 0);
Review Comment:
warning: use nullptr [modernize-use-nullptr]
```suggestion
size_t* total_length = nullptr);
```
##########
be/src/io/fs/stream_load_pipe.h:
##########
@@ -74,7 +74,8 @@ class StreamLoadPipe : public MessageBodySink, public FileReader {
// called when producer/consumer failed
virtual void cancel(const std::string& reason) override;
- Status read_one_message(std::unique_ptr<uint8_t[]>* data, size_t* length);
+ Status read_one_message(std::unique_ptr<uint8_t[]>* data, size_t* length,
+ size_t* total_length = 0);
Review Comment:
warning: use nullptr [modernize-use-nullptr]
```suggestion
size_t* total_length = nullptr);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816527928
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #27145:
URL: https://github.com/apache/doris/pull/27145#discussion_r1432175597
##########
regression-test/suites/load_p0/stream_load/test_json_load.groovy:
##########
@@ -794,7 +793,20 @@ suite("test_json_load", "p0") {
assertEquals("${reason}", "${out}")
}
}
+ } finally {
+ try_sql("DROP TABLE IF EXISTS ${testTable}")
+ }
+
+ // iterate read json when read_json_by_line = false
+ try {
+ sql "DROP TABLE IF EXISTS ${testTable}"
Review Comment:
to make sure the simdjson exception is handled correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1862630830
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit ba7e643a9cb65289151fa2cc41befdc1d333f396, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4689 4383 4469 4383
q2 365 151 158 151
q3 1461 1263 1208 1208
q4 1101 974 943 943
q5 3174 3192 3182 3182
q6 246 126 124 124
q7 1002 488 474 474
q8 2194 2209 2171 2171
q9 6686 6698 6683 6683
q10 3204 3239 3279 3239
q11 311 190 180 180
q12 344 204 200 200
q13 4570 3835 3834 3834
q14 242 220 218 218
q15 572 525 525 525
q16 445 387 383 383
q17 999 574 531 531
q18 7178 6900 6942 6900
q19 1523 1346 1429 1346
q20 557 302 300 300
q21 3070 2627 2660 2627
q22 347 276 286 276
Total cold run time: 44280 ms
Total hot run time: 39878 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4361 4353 4332 4332
q2 265 159 172 159
q3 3548 3523 3521 3521
q4 2376 2370 2367 2367
q5 5768 5768 5757 5757
q6 238 119 119 119
q7 2380 1886 1888 1886
q8 3520 3520 3529 3520
q9 9095 9013 9006 9006
q10 3924 3984 3997 3984
q11 490 359 376 359
q12 762 590 581 581
q13 4316 3597 3599 3597
q14 288 249 260 249
q15 569 535 524 524
q16 506 501 464 464
q17 1878 1848 1871 1848
q18 8621 8244 8316 8244
q19 1735 1755 1746 1746
q20 2282 1963 1944 1944
q21 6513 6199 6140 6140
q22 493 422 414 414
Total cold run time: 63928 ms
Total hot run time: 60761 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1862434726
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.83 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17184123606 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855792192
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855794399
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1817334762
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1817359546
TeamCity be ut coverage result:
Function Coverage: 36.75% (8432/22946)
Line Coverage: 29.25% (68587/234499)
Region Coverage: 27.86% (35458/127294)
Branch Coverage: 24.58% (18089/73578)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8e32fad60e763d2c0ae7f22c6fef0e314ea63382_8e32fad60e763d2c0ae7f22c6fef0e314ea63382/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1817372950
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.76 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098936913 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1815998660
TeamCity be ut coverage result:
Function Coverage: 36.74% (8413/22898)
Line Coverage: 29.26% (68434/233898)
Region Coverage: 27.84% (35375/127060)
Branch Coverage: 24.60% (18070/73468)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8e38c25e0d50b56a352a12d00caf3fe542e6492a_8e38c25e0d50b56a352a12d00caf3fe542e6492a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816553763
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1814481447
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1962310016
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1849570223
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1859180018
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1857220691
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.02 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17217922083 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1859189083
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816019468
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.52 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17099573475 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816732634
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 6879db1185649e1f28f7933c39301ffd78cdffeb, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4966 4698 4707 4698
q2 362 145 158 145
q3 2037 1899 1891 1891
q4 1381 1235 1230 1230
q5 3973 3866 3960 3866
q6 250 133 132 132
q7 1429 890 890 890
q8 2774 2782 2765 2765
q9 9523 19371 9494 9494
q10 3440 3492 3498 3492
q11 379 242 250 242
q12 450 291 297 291
q13 4553 3838 3776 3776
q14 309 294 285 285
q15 574 528 532 528
q16 664 586 582 582
q17 1137 982 921 921
q18 7810 7356 7445 7356
q19 1694 1693 1674 1674
q20 523 319 286 286
q21 4457 4000 4046 4000
q22 480 368 377 368
Total cold run time: 53165 ms
Total hot run time: 48912 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4569 4556 4620 4556
q2 345 224 263 224
q3 4014 4025 3990 3990
q4 2714 2711 2709 2709
q5 9629 9575 9652 9575
q6 245 121 124 121
q7 3036 2509 2491 2491
q8 4478 4442 4439 4439
q9 12865 12824 12753 12753
q10 4070 4182 4150 4150
q11 777 721 651 651
q12 982 815 824 815
q13 4307 3545 3523 3523
q14 373 370 356 356
q15 575 517 518 517
q16 731 664 659 659
q17 3893 3885 3838 3838
q18 9626 9053 9092 9053
q19 1828 1750 1765 1750
q20 2373 2054 2019 2019
q21 8713 8762 8614 8614
q22 874 759 746 746
Total cold run time: 81017 ms
Total hot run time: 77549 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1837753044
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1837948918
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1962310244
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1953982917
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1814490352
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1814730927
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 8b468bcb5170c80a8af43123c05a08d02e27b480, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4950 4696 4656 4656
q2 353 161 159 159
q3 2018 1897 1876 1876
q4 1392 1263 1229 1229
q5 3951 3939 4090 3939
q6 260 128 130 128
q7 1394 876 879 876
q8 2781 2787 2763 2763
q9 9698 9648 9630 9630
q10 3473 3506 3540 3506
q11 370 252 248 248
q12 438 295 303 295
q13 4545 3823 3815 3815
q14 336 290 288 288
q15 587 556 521 521
q16 663 581 576 576
q17 1133 962 955 955
q18 7718 7304 7427 7304
q19 1676 1682 1682 1682
q20 529 315 298 298
q21 4456 3949 4018 3949
q22 481 366 367 366
Total cold run time: 53202 ms
Total hot run time: 49059 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4625 4608 4566 4566
q2 335 248 276 248
q3 4046 4009 3998 3998
q4 2721 2706 2714 2706
q5 9712 9671 9686 9671
q6 251 123 124 123
q7 2601 2311 2238 2238
q8 4421 4438 4425 4425
q9 13196 13105 13188 13105
q10 4083 4191 4195 4191
q11 811 662 653 653
q12 985 805 807 805
q13 4309 3562 3568 3562
q14 390 361 366 361
q15 566 522 519 519
q16 758 663 658 658
q17 3849 3849 3787 3787
q18 9445 8991 8998 8991
q19 1831 1768 1773 1768
q20 2401 2049 2028 2028
q21 8943 8536 8594 8536
q22 880 825 810 810
Total cold run time: 81159 ms
Total hot run time: 77749 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816675719
TeamCity be ut coverage result:
Function Coverage: 36.76% (8429/22927)
Line Coverage: 29.28% (68593/234291)
Region Coverage: 27.86% (35462/127264)
Branch Coverage: 24.59% (18096/73580)
Coverage Report: http://coverage.selectdb-in.cc/coverage/065080943cb0bc531821fc96d5718aa1f8ccfb04_065080943cb0bc531821fc96d5718aa1f8ccfb04/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816012497
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 8e38c25e0d50b56a352a12d00caf3fe542e6492a, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4971 4693 4679 4679
q2 364 152 158 152
q3 2036 1946 1883 1883
q4 1377 1249 1273 1249
q5 3983 3972 4028 3972
q6 243 134 136 134
q7 1379 853 879 853
q8 2778 2829 2797 2797
q9 9837 9793 9658 9658
q10 3447 3534 3535 3534
q11 371 250 251 250
q12 440 295 295 295
q13 4564 3799 3775 3775
q14 323 294 298 294
q15 586 529 524 524
q16 672 590 585 585
q17 1158 945 940 940
q18 7895 7425 7339 7339
q19 1681 1679 1674 1674
q20 522 311 298 298
q21 4435 3990 3965 3965
q22 483 382 375 375
Total cold run time: 53545 ms
Total hot run time: 49225 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4613 4585 4594 4585
q2 347 220 252 220
q3 4043 3994 4007 3994
q4 2714 2714 2693 2693
q5 9739 9594 9648 9594
q6 242 123 125 123
q7 2588 2216 2290 2216
q8 4465 4480 4489 4480
q9 13222 13132 13072 13072
q10 4090 4203 4226 4203
q11 788 639 681 639
q12 977 806 813 806
q13 4282 3553 3585 3553
q14 384 349 348 348
q15 569 520 533 520
q16 746 660 649 649
q17 3854 3858 3804 3804
q18 9658 9026 9139 9026
q19 1824 1800 1785 1785
q20 2401 2067 2067 2067
q21 8815 8419 8434 8419
q22 928 812 824 812
Total cold run time: 81289 ms
Total hot run time: 77608 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1816748924
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.87 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17100227694 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1817333334
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1869986690
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1859178241
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1859195691
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 9eba1e41d2c7b70875ef13ffa095fc30d0fee82d, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4751 4449 4519 4449
q2 358 155 164 155
q3 1466 1216 1174 1174
q4 1100 936 896 896
q5 3148 3146 3164 3146
q6 248 128 122 122
q7 997 482 489 482
q8 2186 2220 2196 2196
q9 6742 6737 6656 6656
q10 3221 3284 3320 3284
q11 316 203 203 203
q12 348 220 209 209
q13 4551 3829 3789 3789
q14 243 210 213 210
q15 561 530 529 529
q16 438 377 373 373
q17 1020 688 566 566
q18 7167 6970 7032 6970
q19 1520 1441 1383 1383
q20 516 291 292 291
q21 3046 2664 2636 2636
q22 349 284 284 284
Total cold run time: 44292 ms
Total hot run time: 40003 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4440 4431 4413 4413
q2 271 161 174 161
q3 3525 3511 3502 3502
q4 2372 2366 2358 2358
q5 5722 5716 5724 5716
q6 242 120 121 120
q7 2380 1904 1826 1826
q8 3530 3538 3546 3538
q9 9030 9006 8964 8964
q10 3910 4018 3993 3993
q11 504 386 389 386
q12 764 600 611 600
q13 4285 3540 3570 3540
q14 281 257 255 255
q15 569 516 517 516
q16 503 450 443 443
q17 1879 1879 1857 1857
q18 8700 8270 8301 8270
q19 1739 1769 1764 1764
q20 2243 1959 1941 1941
q21 6501 6222 6149 6149
q22 511 418 424 418
Total cold run time: 63901 ms
Total hot run time: 60730 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](json_reader) fix simd json reader lose data when parse [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1855794500
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd_json_reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1862416584
TeamCity be ut coverage result:
Function Coverage: 36.49% (8533/23387)
Line Coverage: 28.62% (69357/242311)
Region Coverage: 27.64% (35890/129860)
Branch Coverage: 24.36% (18336/75258)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ba7e643a9cb65289151fa2cc41befdc1d333f396_ba7e643a9cb65289151fa2cc41befdc1d333f396/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](simd-json-reader) fix simd json reader lose data and support stream parser [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27145:
URL: https://github.com/apache/doris/pull/27145#issuecomment-1954057385
<details>
<summary>TPC-H: <b>Total hot run time: 41174 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6d5c3bc10708aedb7ba5aff316e5f6e38039abb, data reload: false
------ Round 1 ----------------------------------
q1 17673 5001 4874 4874
q2 2056 148 135 135
q3 10577 1006 997 997
q4 4661 958 960 958
q5 7728 3182 3249 3182
q6 199 135 134 134
q7 1257 789 758 758
q8 9312 2075 2066 2066
q9 7536 6672 6616 6616
q10 8310 2631 2628 2628
q11 412 218 219 218
q12 760 332 331 331
q13 18000 3668 3636 3636
q14 291 257 264 257
q15 579 549 502 502
q16 485 401 413 401
q17 919 862 802 802
q18 7414 6546 6634 6546
q19 1560 1490 1486 1486
q20 641 336 333 333
q21 7334 4022 3965 3965
q22 874 349 352 349
Total cold run time: 108578 ms
Total hot run time: 41174 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4810 4819 4874 4819
q2 295 185 191 185
q3 3603 3586 3557 3557
q4 2517 2536 2525 2525
q5 5737 5724 5730 5724
q6 215 129 123 123
q7 2240 1644 1694 1644
q8 2978 3091 3074 3074
q9 8633 8620 8638 8620
q10 6774 4230 4215 4215
q11 516 395 399 395
q12 773 548 539 539
q13 4391 3418 3426 3418
q14 269 237 223 223
q15 639 543 551 543
q16 480 426 431 426
q17 1670 1606 1624 1606
q18 8329 7586 7710 7586
q19 1631 1626 1623 1623
q20 2106 1845 1833 1833
q21 6572 6191 6165 6165
q22 577 502 496 496
Total cold run time: 65755 ms
Total hot run time: 59339 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org