You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/11/20 06:34:46 UTC

[PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

HHoflittlefish777 opened a new pull request, #27272:
URL: https://github.com/apache/doris/pull/27272

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826780995

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit d47bd7f70baaec9ebce4aab6ed3030d46ca08bb9, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4893	4660	4638	4638
   q2	357	157	157	157
   q3	2070	1900	1912	1900
   q4	1387	1262	1233	1233
   q5	3944	3966	4013	3966
   q6	250	131	131	131
   q7	1436	888	882	882
   q8	2771	2787	2789	2787
   q9	9541	9521	9605	9521
   q10	3490	3525	3513	3513
   q11	369	241	246	241
   q12	437	288	289	288
   q13	4563	3817	3832	3817
   q14	310	286	290	286
   q15	593	540	516	516
   q16	687	579	594	579
   q17	1142	987	962	962
   q18	7748	7383	7328	7328
   q19	1699	1697	1671	1671
   q20	524	322	302	302
   q21	4389	3948	3994	3948
   q22	481	370	370	370
   Total cold run time: 53081 ms
   Total hot run time: 49036 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4599	4611	4590	4590
   q2	352	216	265	216
   q3	4033	4005	3991	3991
   q4	2694	2672	2685	2672
   q5	9523	9611	9583	9583
   q6	246	125	126	125
   q7	3006	2454	2482	2454
   q8	4469	4427	4490	4427
   q9	12952	12808	12828	12808
   q10	4073	4191	4159	4159
   q11	785	667	662	662
   q12	976	812	809	809
   q13	4270	3570	3581	3570
   q14	381	362	356	356
   q15	591	530	523	523
   q16	736	687	670	670
   q17	3876	3842	3889	3842
   q18	9540	8989	9076	8989
   q19	1821	1781	1780	1780
   q20	2454	2068	2077	2068
   q21	8779	8757	8538	8538
   q22	869	837	861	837
   Total cold run time: 81025 ms
   Total hot run time: 77669 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825190598

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit fc2752ab96d92b5ec5a2551269472bdc716bf746, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4878	4675	4652	4652
   q2	360	141	159	141
   q3	2028	1904	1833	1833
   q4	1431	1241	1226	1226
   q5	3937	3943	3987	3943
   q6	247	130	133	130
   q7	1426	889	897	889
   q8	2776	2777	2774	2774
   q9	9752	9568	9422	9422
   q10	3465	3500	3503	3500
   q11	394	251	249	249
   q12	440	296	293	293
   q13	4558	3775	3778	3775
   q14	329	292	290	290
   q15	583	541	519	519
   q16	661	582	583	582
   q17	1123	953	910	910
   q18	7838	7291	7381	7291
   q19	1651	1663	1672	1663
   q20	536	301	315	301
   q21	4427	4002	3940	3940
   q22	484	367	369	367
   Total cold run time: 53324 ms
   Total hot run time: 48690 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4569	4573	4578	4573
   q2	342	219	268	219
   q3	4023	3995	4005	3995
   q4	2695	2701	2682	2682
   q5	9685	9780	9796	9780
   q6	244	122	122	122
   q7	3024	2468	2451	2451
   q8	4451	4439	4463	4439
   q9	12851	12760	12772	12760
   q10	4114	4183	4167	4167
   q11	773	622	629	622
   q12	986	814	812	812
   q13	4245	3579	3539	3539
   q14	377	340	337	337
   q15	562	517	520	517
   q16	739	665	672	665
   q17	3850	3816	3841	3816
   q18	9550	9014	8984	8984
   q19	1781	1757	1771	1757
   q20	2386	2068	2028	2028
   q21	8790	8561	8622	8561
   q22	845	756	810	756
   Total cold run time: 80882 ms
   Total hot run time: 77582 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818774687

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826771282

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818825171

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.01 seconds
    stream load tsv:          578 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.6 seconds inserted 10000000 Rows, about 349K ops/s
    storage size: 17158949123 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825122993

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825120070

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1824332270

   @eldenmoon would you please take a look at this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826783637

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.91 seconds
    stream load tsv:          565 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         27 seconds loaded 2358488459 Bytes, about 83 MB/s
    stream load orc:          71 seconds loaded 1101869774 Bytes, about 14 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.6 seconds inserted 10000000 Rows, about 349K ops/s
    storage size: 17098855526 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818883709

   TeamCity be ut coverage result:
    Function Coverage: 36.51% (8448/23137) 
    Line Coverage: 28.86% (68663/237917)
    Region Coverage: 27.83% (35507/127597)
    Branch Coverage: 24.59% (18105/73632)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/5558f97503295f591a0b3904e767b652a042aee9_5558f97503295f591a0b3904e767b652a042aee9/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818819885

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 5558f97503295f591a0b3904e767b652a042aee9, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4954	4667	4680	4667
   q2	355	159	159	159
   q3	2054	1904	1888	1888
   q4	1415	1268	1257	1257
   q5	3977	3939	4046	3939
   q6	254	127	129	127
   q7	1432	884	898	884
   q8	2745	2806	2776	2776
   q9	9845	9723	9533	9533
   q10	3458	3553	3536	3536
   q11	376	242	249	242
   q12	443	296	302	296
   q13	4589	3828	3810	3810
   q14	317	286	287	286
   q15	590	540	515	515
   q16	668	591	574	574
   q17	1145	954	943	943
   q18	7820	7501	7358	7358
   q19	1666	1679	1689	1679
   q20	578	325	316	316
   q21	4387	3975	3985	3975
   q22	478	371	381	371
   Total cold run time: 53546 ms
   Total hot run time: 49131 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4602	4580	4590	4580
   q2	337	239	270	239
   q3	4031	4005	3987	3987
   q4	2716	2696	2700	2696
   q5	9703	9601	9669	9601
   q6	245	124	125	124
   q7	3024	2466	2441	2441
   q8	4450	4444	4460	4444
   q9	13220	13149	13233	13149
   q10	4104	4196	4193	4193
   q11	786	662	652	652
   q12	984	817	812	812
   q13	4264	3587	3605	3587
   q14	378	336	354	336
   q15	580	520	530	520
   q16	747	668	658	658
   q17	3915	3870	3905	3870
   q18	9554	9064	9099	9064
   q19	1824	1778	1773	1773
   q20	2415	2076	2049	2049
   q21	8843	8536	8794	8536
   q22	886	864	804	804
   Total cold run time: 81608 ms
   Total hot run time: 78115 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1829939771

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818314237

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818764824

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1826772606

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825246396

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #27272:
URL: https://github.com/apache/doris/pull/27272#discussion_r1403402330


##########
be/src/vec/exec/format/json/new_json_reader.cpp:
##########
@@ -1437,48 +1459,38 @@ Status NewJsonReader::_append_error_msg(simdjson::ondemand::object* obj, std::st
     return Status::OK();
 }
 
-Status NewJsonReader::_simdjson_parse_json(bool* is_empty_row, bool* eof) {
-    size_t size = 0;
-    RETURN_IF_ERROR(_simdjson_parse_json_doc(&size, eof));
-
-    // read all data, then return
-    if (size == 0 || *eof) {
-        *is_empty_row = true;
+Status NewJsonReader::_simdjson_parse_json(size_t* size, bool* is_empty_row, bool* eof,
+                                           simdjson::error_code* error) {
+    SCOPED_TIMER(_file_read_timer);
+    // step1: read buf from pipe.
+    RETURN_IF_ERROR(_read_pipe_buf(size, eof));
+    if (*eof) {
         return Status::OK();
     }
 
-    if (!_parsed_jsonpaths.empty() && _strip_outer_array) {
-        _total_rows = _json_value.count_elements().value();
-        _next_row = 0;
-
-        if (_total_rows == 0) {
-            // meet an empty json array.
-            *is_empty_row = true;
-        }
-    }
+    // step2: init json parser iterate.
+    RETURN_IF_ERROR(_init_json_parser_iterate(size, error));
     return Status::OK();
 }
-Status NewJsonReader::_simdjson_parse_json_doc(size_t* size, bool* eof) {
-    // read a whole message
-    SCOPED_TIMER(_file_read_timer);
-    const uint8_t* json_str = nullptr;
-    std::unique_ptr<uint8_t[]> json_str_ptr;
+
+Status NewJsonReader::_read_pipe_buf(size_t* size, bool* eof) {

Review Comment:
   the function name `read_pipe_buf` is not suitable here for the data source is not necessarily a pipe.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818308292

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818454950

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit b671708caf63fb85e350d6288fb15396a3d5e7d7, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4914	4651	4648	4648
   q2	356	161	159	159
   q3	2041	1888	1889	1888
   q4	1381	1259	1261	1259
   q5	3983	3970	4043	3970
   q6	250	129	125	125
   q7	1395	893	910	893
   q8	2760	2802	2787	2787
   q9	9747	9812	9623	9623
   q10	3462	3549	3542	3542
   q11	379	252	246	246
   q12	442	289	297	289
   q13	4574	3864	3828	3828
   q14	318	298	274	274
   q15	584	533	524	524
   q16	670	581	578	578
   q17	1134	981	960	960
   q18	7835	7412	7327	7327
   q19	1684	1680	1683	1680
   q20	520	319	299	299
   q21	4389	3944	3959	3944
   q22	464	370	373	370
   Total cold run time: 53282 ms
   Total hot run time: 49213 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4598	4582	4569	4569
   q2	339	246	247	246
   q3	3999	4006	3992	3992
   q4	2693	2703	2698	2698
   q5	9601	9581	9535	9535
   q6	243	123	122	122
   q7	3027	2475	2487	2475
   q8	4471	4451	4474	4451
   q9	13279	13216	13128	13128
   q10	4118	4173	4174	4173
   q11	817	663	651	651
   q12	976	813	821	813
   q13	4314	3571	3657	3571
   q14	378	350	356	350
   q15	584	521	526	521
   q16	726	679	669	669
   q17	3780	3923	3835	3835
   q18	9521	8916	9109	8916
   q19	1810	1802	1806	1802
   q20	2430	2064	2073	2064
   q21	8778	8663	8656	8656
   q22	840	788	812	788
   Total cold run time: 81322 ms
   Total hot run time: 78025 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](json_reader) refactor json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1818351154

   TeamCity be ut coverage result:
    Function Coverage: 36.52% (8450/23137) 
    Line Coverage: 28.88% (68691/237888)
    Region Coverage: 27.84% (35517/127564)
    Branch Coverage: 24.61% (18113/73612)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/b671708caf63fb85e350d6288fb15396a3d5e7d7_b671708caf63fb85e350d6288fb15396a3d5e7d7/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825246440

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #27272:
URL: https://github.com/apache/doris/pull/27272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](simd_json_reader) refactor simd json reader to adapt to parse multi json [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27272:
URL: https://github.com/apache/doris/pull/27272#issuecomment-1825188684

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.54 seconds
    stream load tsv:          571 seconds loaded 74807831229 Bytes, about 124 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          31.9 seconds inserted 10000000 Rows, about 313K ops/s
    storage size: 17098972618 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org