You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/01/25 10:14:18 UTC

[PR] [feature](journal) Add a method to write a set of journals in batch [doris]

w41ter opened a new pull request, #30380:
URL: https://github.com/apache/doris/pull/30380

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #30380:
URL: https://github.com/apache/doris/pull/30380


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1909825348

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1909926004

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.86 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 46825895e2ed5d4c9496aa48cc24c6f460006508, data reload: false
   
   query1	0.03	0.03	0.02
   query2	0.06	0.02	0.03
   query3	0.22	0.05	0.05
   query4	1.69	0.08	0.07
   query5	0.53	0.53	0.53
   query6	1.28	0.64	0.64
   query7	0.02	0.01	0.02
   query8	0.03	0.03	0.02
   query9	0.54	0.50	0.50
   query10	0.55	0.55	0.55
   query11	0.12	0.08	0.09
   query12	0.12	0.09	0.08
   query13	0.59	0.62	0.61
   query14	0.79	0.80	0.80
   query15	0.79	0.78	0.78
   query16	0.38	0.36	0.36
   query17	1.03	1.00	1.01
   query18	0.25	0.24	0.24
   query19	1.88	1.79	1.79
   query20	0.01	0.01	0.02
   query21	15.39	0.58	0.56
   query22	2.56	2.10	1.76
   query23	17.43	1.09	0.80
   query24	2.73	0.88	0.94
   query25	0.37	0.23	0.17
   query26	0.46	0.14	0.13
   query27	0.06	0.06	0.05
   query28	11.69	0.78	0.79
   query29	12.49	3.23	3.17
   query30	0.52	0.48	0.48
   query31	2.78	0.34	0.35
   query32	3.36	0.48	0.49
   query33	3.23	3.29	3.26
   query34	15.80	4.34	4.27
   query35	4.34	4.35	4.24
   query36	1.11	1.06	1.07
   query37	0.06	0.06	0.05
   query38	0.04	0.03	0.03
   query39	0.03	0.02	0.01
   query40	0.16	0.14	0.14
   query41	0.08	0.01	0.02
   query42	0.02	0.01	0.01
   query43	0.02	0.02	0.02
   Total cold run time: 105.64 s
   Total hot run time: 30.86 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1910207891

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1909899616

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38727 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 46825895e2ed5d4c9496aa48cc24c6f460006508, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17658	5250	5149	5149
   q2	2037	137	133	133
   q3	10639	1124	1176	1124
   q4	10227	772	779	772
   q5	7721	3124	3136	3124
   q6	193	122	125	122
   q7	867	492	485	485
   q8	9205	1915	1936	1915
   q9	7270	6386	6334	6334
   q10	8237	3026	3031	3026
   q11	425	210	198	198
   q12	350	188	190	188
   q13	18004	3354	3348	3348
   q14	239	207	211	207
   q15	542	508	505	505
   q16	489	369	378	369
   q17	937	544	577	544
   q18	7396	6958	6810	6810
   q19	1545	1403	1402	1402
   q20	590	301	303	301
   q21	2742	2410	2377	2377
   q22	364	294	305	294
   Total cold run time: 107677 ms
   Total hot run time: 38727 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5300	5354	5247	5247
   q2	327	219	212	212
   q3	3362	3195	3236	3195
   q4	2097	2064	2066	2064
   q5	5853	5903	5955	5903
   q6	200	114	118	114
   q7	2369	1825	1816	1816
   q8	3210	3371	3367	3367
   q9	8863	8864	8895	8864
   q10	3942	3848	3836	3836
   q11	558	489	444	444
   q12	837	627	634	627
   q13	16921	3186	3138	3138
   q14	290	253	273	253
   q15	543	514	492	492
   q16	498	451	439	439
   q17	3668	1836	1861	1836
   q18	9597	17763	9729	9729
   q19	23171	1555	1541	1541
   q20	4608	1936	1933	1933
   q21	14565	5381	5520	5381
   q22	976	542	509	509
   Total cold run time: 111755 ms
   Total hot run time: 60940 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1909917773

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 187046 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 46825895e2ed5d4c9496aa48cc24c6f460006508, data reload: false
   
   query1	939	345	334	334
   query2	6528	1951	1934	1934
   query3	6697	203	202	202
   query4	32815	22370	22155	22155
   query5	4454	452	379	379
   query6	244	162	149	149
   query7	4609	271	267	267
   query8	230	171	174	171
   query9	8320	2640	2626	2626
   query10	419	227	216	216
   query11	16913	15498	15564	15498
   query12	123	72	67	67
   query13	1679	373	392	373
   query14	10605	7148	7070	7070
   query15	214	185	187	185
   query16	5792	260	248	248
   query17	941	477	475	475
   query18	1795	261	248	248
   query19	182	131	136	131
   query20	67	70	67	67
   query21	180	134	130	130
   query22	4904	4809	4800	4800
   query23	32263	30988	31081	30988
   query24	5316	2867	2865	2865
   query25	407	306	315	306
   query26	791	145	157	145
   query27	2250	287	291	287
   query28	3766	1860	1832	1832
   query29	890	622	626	622
   query30	195	140	138	138
   query31	904	771	772	771
   query32	63	51	50	50
   query33	376	216	215	215
   query34	811	478	472	472
   query35	905	809	804	804
   query36	1211	1178	1237	1178
   query37	98	58	61	58
   query38	3383	3249	3203	3203
   query39	1324	1268	1268	1268
   query40	188	86	80	80
   query41	36	40	34	34
   query42	88	78	90	78
   query43	502	478	449	449
   query44	1066	701	710	701
   query45	194	178	178	178
   query46	1047	658	668	658
   query47	1690	1494	1505	1494
   query48	392	307	308	307
   query49	623	292	293	292
   query50	679	318	310	310
   query51	5331	5245	5245	5245
   query52	90	74	76	74
   query53	330	261	260	260
   query54	216	181	182	181
   query55	76	82	79	79
   query56	171	178	173	173
   query57	1021	910	931	910
   query58	169	161	161	161
   query59	2787	2775	2597	2597
   query60	212	183	181	181
   query61	80	87	79	79
   query62	448	375	353	353
   query63	288	259	268	259
   query64	2547	1822	1801	1801
   query65	3350	3305	3268	3268
   query66	1090	314	310	310
   query67	15541	14847	15165	14847
   query68	12557	525	507	507
   query69	612	312	308	308
   query70	1670	1521	1516	1516
   query71	10449	10204	10216	10204
   query72	5063	2805	2825	2805
   query73	2629	313	307	307
   query74	6965	6337	6337	6337
   query75	4912	2352	2300	2300
   query76	6511	1027	1029	1027
   query77	692	232	236	232
   query78	9634	8627	8611	8611
   query79	1018	495	511	495
   query80	606	317	323	317
   query81	476	210	201	201
   query82	200	79	78	78
   query83	136	118	119	118
   query84	275	68	68	68
   query85	1080	326	318	318
   query86	388	376	386	376
   query87	3481	3321	3336	3321
   query88	3123	2197	2203	2197
   query89	420	371	366	366
   query90	2256	192	191	191
   query91	155	123	120	120
   query92	56	43	46	43
   query93	1519	422	444	422
   query94	1462	163	156	156
   query95	506	467	463	463
   query96	646	320	325	320
   query97	4229	4146	4174	4146
   query98	206	188	184	184
   query99	959	706	710	706
   Total cold run time: 288746 ms
   Total hot run time: 187046 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1910305856

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](journal) Add a method to write a set of journals in batch [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30380:
URL: https://github.com/apache/doris/pull/30380#issuecomment-1909930647

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 46825895e2ed5d4c9496aa48cc24c6f460006508 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       14.8 seconds inserted 10000000 Rows, about 675K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org