You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/12/11 09:30:45 UTC

[PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Mryange opened a new pull request, #28248:
URL: https://github.com/apache/doris/pull/28248

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849865890

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1851066294

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1851095239

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit a51ef03bb7ac5b147141fe3cba2b91cd07ac4783, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4733	4480	4708	4480
   q2	363	153	162	153
   q3	1446	1203	1195	1195
   q4	1112	952	874	874
   q5	3120	3120	3139	3120
   q6	250	127	127	127
   q7	979	495	479	479
   q8	2226	2208	2190	2190
   q9	6703	6696	6669	6669
   q10	3221	3277	3300	3277
   q11	325	206	198	198
   q12	343	211	211	211
   q13	4563	3863	3814	3814
   q14	242	208	206	206
   q15	563	521	527	521
   q16	455	383	389	383
   q17	1011	585	534	534
   q18	7546	7094	7103	7094
   q19	1521	1311	1463	1311
   q20	533	300	320	300
   q21	3038	2660	2670	2660
   q22	351	277	279	277
   Total cold run time: 44644 ms
   Total hot run time: 40073 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4413	4400	4381	4381
   q2	267	164	171	164
   q3	3552	3557	3537	3537
   q4	2392	2377	2380	2377
   q5	5745	5735	5760	5735
   q6	243	121	122	121
   q7	2416	1872	1912	1872
   q8	3536	3519	3530	3519
   q9	9052	9000	8971	8971
   q10	3885	4009	4006	4006
   q11	494	393	373	373
   q12	763	598	594	594
   q13	4314	3573	3554	3554
   q14	289	253	252	252
   q15	569	518	517	517
   q16	516	468	476	468
   q17	1891	1860	1862	1860
   q18	8808	8332	8373	8332
   q19	1794	1763	1741	1741
   q20	2265	1946	1935	1935
   q21	6538	6199	6155	6155
   q22	496	413	437	413
   Total cold run time: 64238 ms
   Total hot run time: 60877 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849648159

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849678386

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1850151504

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit a51ef03bb7ac5b147141fe3cba2b91cd07ac4783, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4711	4506	4457	4457
   q2	363	147	157	147
   q3	1456	1270	1211	1211
   q4	1098	936	843	843
   q5	3139	3162	3173	3162
   q6	254	127	128	127
   q7	1009	491	474	474
   q8	2207	2229	2188	2188
   q9	6650	6686	6647	6647
   q10	3241	3260	3255	3255
   q11	331	194	194	194
   q12	347	200	205	200
   q13	4632	4258	3797	3797
   q14	241	215	209	209
   q15	574	519	521	519
   q16	444	385	384	384
   q17	1001	554	541	541
   q18	7601	7531	7827	7531
   q19	1517	1333	1452	1333
   q20	546	310	285	285
   q21	3074	2648	2738	2648
   q22	342	273	286	273
   Total cold run time: 44778 ms
   Total hot run time: 40425 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4402	4409	4394	4394
   q2	266	160	173	160
   q3	3553	3547	3538	3538
   q4	2388	2376	2375	2375
   q5	5753	5760	5752	5752
   q6	241	121	120	120
   q7	2372	1904	1877	1877
   q8	3533	3536	3527	3527
   q9	9030	9012	8944	8944
   q10	3890	4008	3991	3991
   q11	494	380	376	376
   q12	763	588	618	588
   q13	4312	3552	3568	3552
   q14	283	258	259	258
   q15	566	525	513	513
   q16	502	450	472	450
   q17	1884	1840	1856	1840
   q18	8763	8273	8301	8273
   q19	1731	1735	1723	1723
   q20	2263	1942	1920	1920
   q21	6561	6213	6222	6213
   q22	513	412	415	412
   Total cold run time: 64063 ms
   Total hot run time: 60796 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849933787

   run pipelinex_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849650153

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849755931

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit a51ef03bb7ac5b147141fe3cba2b91cd07ac4783, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4598	4450	4481	4450
   q2	362	148	162	148
   q3	1450	1205	1221	1205
   q4	1113	942	904	904
   q5	3119	3127	3132	3127
   q6	250	128	125	125
   q7	1006	486	471	471
   q8	2218	2236	2183	2183
   q9	6701	7205	6661	6661
   q10	3199	3258	3253	3253
   q11	320	208	207	207
   q12	352	213	211	211
   q13	4640	4011	3824	3824
   q14	239	216	222	216
   q15	563	525	523	523
   q16	440	392	397	392
   q17	1024	594	564	564
   q18	8351	7585	7888	7585
   q19	1522	1375	1460	1375
   q20	541	315	287	287
   q21	3085	2655	2686	2655
   q22	349	282	283	282
   Total cold run time: 45442 ms
   Total hot run time: 40648 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4404	4381	4415	4381
   q2	266	164	166	164
   q3	3551	3559	3529	3529
   q4	2377	2383	2379	2379
   q5	5743	5744	5740	5740
   q6	241	121	121	121
   q7	2429	1909	1879	1879
   q8	3524	3526	3542	3526
   q9	9079	8985	8999	8985
   q10	3918	3971	3997	3971
   q11	506	391	379	379
   q12	771	586	587	586
   q13	4283	3569	3537	3537
   q14	291	254	259	254
   q15	576	509	518	509
   q16	503	439	464	439
   q17	1861	1843	1847	1843
   q18	8790	8528	8348	8348
   q19	1786	1807	1786	1786
   q20	2284	1943	1935	1935
   q21	6572	6194	6171	6171
   q22	498	419	416	416
   Total cold run time: 64253 ms
   Total hot run time: 60878 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28248:
URL: https://github.com/apache/doris/pull/28248#discussion_r1422190889


##########
be/src/pipeline/exec/partition_sort_sink_operator.h:
##########
@@ -105,6 +105,12 @@ class PartitionSortSinkOperatorX final : public DataSinkOperatorX<PartitionSortS
     Status open(RuntimeState* state) override;
     Status sink(RuntimeState* state, vectorized::Block* in_block,
                 SourceState source_state) override;
+    ExchangeType get_local_exchange_type() const override {

Review Comment:
   warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
       static ExchangeType get_local_exchange_type() override {
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange closed pull request #28248: [only test](pipelineX) add local_shuffle in partition sort node
URL: https://github.com/apache/doris/pull/28248


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1850101884

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849768349

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.24 seconds
    stream load tsv:          580 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          34 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
    storage size: 17222925735 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1850164479

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 43.86 seconds
    stream load tsv:          577 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.7 seconds inserted 10000000 Rows, about 348K ops/s
    storage size: 17219793564 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1851105870

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.64 seconds
    stream load tsv:          579 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17218035322 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org