You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/11/20 06:34:46 UTC
[PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
HHoflittlefish777 opened a new pull request, #27272:
URL: https://github.com/apache/doris/pull/27272
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826780995
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit d47bd7f70baaec9ebce4aab6ed3030d46ca08bb9, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4893 4660 4638 4638
q2 357 157 157 157
q3 2070 1900 1912 1900
q4 1387 1262 1233 1233
q5 3944 3966 4013 3966
q6 250 131 131 131
q7 1436 888 882 882
q8 2771 2787 2789 2787
q9 9541 9521 9605 9521
q10 3490 3525 3513 3513
q11 369 241 246 241
q12 437 288 289 288
q13 4563 3817 3832 3817
q14 310 286 290 286
q15 593 540 516 516
q16 687 579 594 579
q17 1142 987 962 962
q18 7748 7383 7328 7328
q19 1699 1697 1671 1671
q20 524 322 302 302
q21 4389 3948 3994 3948
q22 481 370 370 370
Total cold run time: 53081 ms
Total hot run time: 49036 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4599 4611 4590 4590
q2 352 216 265 216
q3 4033 4005 3991 3991
q4 2694 2672 2685 2672
q5 9523 9611 9583 9583
q6 246 125 126 125
q7 3006 2454 2482 2454
q8 4469 4427 4490 4427
q9 12952 12808 12828 12808
q10 4073 4191 4159 4159
q11 785 667 662 662
q12 976 812 809 809
q13 4270 3570 3581 3570
q14 381 362 356 356
q15 591 530 523 523
q16 736 687 670 670
q17 3876 3842 3889 3842
q18 9540 8989 9076 8989
q19 1821 1781 1780 1780
q20 2454 2068 2077 2068
q21 8779 8757 8538 8538
q22 869 837 861 837
Total cold run time: 81025 ms
Total hot run time: 77669 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825190598
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit fc2752ab96d92b5ec5a2551269472bdc716bf746, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4878 4675 4652 4652
q2 360 141 159 141
q3 2028 1904 1833 1833
q4 1431 1241 1226 1226
q5 3937 3943 3987 3943
q6 247 130 133 130
q7 1426 889 897 889
q8 2776 2777 2774 2774
q9 9752 9568 9422 9422
q10 3465 3500 3503 3500
q11 394 251 249 249
q12 440 296 293 293
q13 4558 3775 3778 3775
q14 329 292 290 290
q15 583 541 519 519
q16 661 582 583 582
q17 1123 953 910 910
q18 7838 7291 7381 7291
q19 1651 1663 1672 1663
q20 536 301 315 301
q21 4427 4002 3940 3940
q22 484 367 369 367
Total cold run time: 53324 ms
Total hot run time: 48690 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4569 4573 4578 4573
q2 342 219 268 219
q3 4023 3995 4005 3995
q4 2695 2701 2682 2682
q5 9685 9780 9796 9780
q6 244 122 122 122
q7 3024 2468 2451 2451
q8 4451 4439 4463 4439
q9 12851 12760 12772 12760
q10 4114 4183 4167 4167
q11 773 622 629 622
q12 986 814 812 812
q13 4245 3579 3539 3539
q14 377 340 337 337
q15 562 517 520 517
q16 739 665 672 665
q17 3850 3816 3841 3816
q18 9550 9014 8984 8984
q19 1781 1757 1771 1757
q20 2386 2068 2028 2028
q21 8790 8561 8622 8561
q22 845 756 810 756
Total cold run time: 80882 ms
Total hot run time: 77582 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818774687
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826771282
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818825171
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.01 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17158949123 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825122993
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825120070
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1824332270
@eldenmoon would you please take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826783637
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.91 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 71 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17098855526 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818883709
TeamCity be ut coverage result:
Function Coverage: 36.51% (8448/23137)
Line Coverage: 28.86% (68663/237917)
Region Coverage: 27.83% (35507/127597)
Branch Coverage: 24.59% (18105/73632)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5558f97503295f591a0b3904e767b652a042aee9_5558f97503295f591a0b3904e767b652a042aee9/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818819885
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 5558f97503295f591a0b3904e767b652a042aee9, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4954 4667 4680 4667
q2 355 159 159 159
q3 2054 1904 1888 1888
q4 1415 1268 1257 1257
q5 3977 3939 4046 3939
q6 254 127 129 127
q7 1432 884 898 884
q8 2745 2806 2776 2776
q9 9845 9723 9533 9533
q10 3458 3553 3536 3536
q11 376 242 249 242
q12 443 296 302 296
q13 4589 3828 3810 3810
q14 317 286 287 286
q15 590 540 515 515
q16 668 591 574 574
q17 1145 954 943 943
q18 7820 7501 7358 7358
q19 1666 1679 1689 1679
q20 578 325 316 316
q21 4387 3975 3985 3975
q22 478 371 381 371
Total cold run time: 53546 ms
Total hot run time: 49131 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4602 4580 4590 4580
q2 337 239 270 239
q3 4031 4005 3987 3987
q4 2716 2696 2700 2696
q5 9703 9601 9669 9601
q6 245 124 125 124
q7 3024 2466 2441 2441
q8 4450 4444 4460 4444
q9 13220 13149 13233 13149
q10 4104 4196 4193 4193
q11 786 662 652 652
q12 984 817 812 812
q13 4264 3587 3605 3587
q14 378 336 354 336
q15 580 520 530 520
q16 747 668 658 658
q17 3915 3870 3905 3870
q18 9554 9064 9099 9064
q19 1824 1778 1773 1773
q20 2415 2076 2049 2049
q21 8843 8536 8794 8536
q22 886 864 804 804
Total cold run time: 81608 ms
Total hot run time: 78115 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1829939771
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818314237
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818764824
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826772606
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825246396
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #27272:
URL: https://github.com/apache/doris/pull/27272#discussion_r1403402330
##########
be/src/vec/exec/format/json/new_json_reader.cpp:
##########
@@ -1437,48 +1459,38 @@ Status NewJsonReader::_append_error_msg(simdjson::ondemand::object* obj, std::st
return Status::OK();
}
-Status NewJsonReader::_simdjson_parse_json(bool* is_empty_row, bool* eof) {
- size_t size = 0;
- RETURN_IF_ERROR(_simdjson_parse_json_doc(&size, eof));
-
- // read all data, then return
- if (size == 0 || *eof) {
- *is_empty_row = true;
+Status NewJsonReader::_simdjson_parse_json(size_t* size, bool* is_empty_row, bool* eof,
+ simdjson::error_code* error) {
+ SCOPED_TIMER(_file_read_timer);
+ // step1: read buf from pipe.
+ RETURN_IF_ERROR(_read_pipe_buf(size, eof));
+ if (*eof) {
return Status::OK();
}
- if (!_parsed_jsonpaths.empty() && _strip_outer_array) {
- _total_rows = _json_value.count_elements().value();
- _next_row = 0;
-
- if (_total_rows == 0) {
- // meet an empty json array.
- *is_empty_row = true;
- }
- }
+ // step2: init json parser iterate.
+ RETURN_IF_ERROR(_init_json_parser_iterate(size, error));
return Status::OK();
}
-Status NewJsonReader::_simdjson_parse_json_doc(size_t* size, bool* eof) {
- // read a whole message
- SCOPED_TIMER(_file_read_timer);
- const uint8_t* json_str = nullptr;
- std::unique_ptr<uint8_t[]> json_str_ptr;
+
+Status NewJsonReader::_read_pipe_buf(size_t* size, bool* eof) {
Review Comment:
the function name `read_pipe_buf` is not suitable here for the data source is not necessarily a pipe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818308292
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818454950
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit b671708caf63fb85e350d6288fb15396a3d5e7d7, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4914 4651 4648 4648
q2 356 161 159 159
q3 2041 1888 1889 1888
q4 1381 1259 1261 1259
q5 3983 3970 4043 3970
q6 250 129 125 125
q7 1395 893 910 893
q8 2760 2802 2787 2787
q9 9747 9812 9623 9623
q10 3462 3549 3542 3542
q11 379 252 246 246
q12 442 289 297 289
q13 4574 3864 3828 3828
q14 318 298 274 274
q15 584 533 524 524
q16 670 581 578 578
q17 1134 981 960 960
q18 7835 7412 7327 7327
q19 1684 1680 1683 1680
q20 520 319 299 299
q21 4389 3944 3959 3944
q22 464 370 373 370
Total cold run time: 53282 ms
Total hot run time: 49213 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4598 4582 4569 4569
q2 339 246 247 246
q3 3999 4006 3992 3992
q4 2693 2703 2698 2698
q5 9601 9581 9535 9535
q6 243 123 122 122
q7 3027 2475 2487 2475
q8 4471 4451 4474 4451
q9 13279 13216 13128 13128
q10 4118 4173 4174 4173
q11 817 663 651 651
q12 976 813 821 813
q13 4314 3571 3657 3571
q14 378 350 356 350
q15 584 521 526 521
q16 726 679 669 669
q17 3780 3923 3835 3835
q18 9521 8916 9109 8916
q19 1810 1802 1806 1802
q20 2430 2064 2073 2064
q21 8778 8663 8656 8656
q22 840 788 812 788
Total cold run time: 81322 ms
Total hot run time: 78025 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818351154
TeamCity be ut coverage result:
Function Coverage: 36.52% (8450/23137)
Line Coverage: 28.88% (68691/237888)
Region Coverage: 27.84% (35517/127564)
Branch Coverage: 24.61% (18113/73612)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b671708caf63fb85e350d6288fb15396a3d5e7d7_b671708caf63fb85e350d6288fb15396a3d5e7d7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825246440
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #27272:
URL: https://github.com/apache/doris/pull/27272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825188684
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 31.9 seconds inserted 10000000 Rows, about 313K ops/s
storage size: 17098972618 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org