You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2024/07/10 07:47:19 UTC

[PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Yukang-Lian opened a new pull request, #37598:
URL: https://github.com/apache/doris/pull/37598

   Problem: An error is encountered when executing a schema change on a unique table to add a column with a complex type, such as bitmap, as documented in https://github.com/apache/doris/issues/31365
   
   Reason: The error arises because the schema change logic erroneously applies an aggregation check for new columns across all table types, demanding an explicit aggregation type declaration. However, unique and duplicate tables inherently assume default aggregation types for newly added columns, leading to this misstep.
   
   Solution: The schema change process for introducing new columns needs to distinguish between table types accurately. For unique and duplicate tables, it should automatically assign the appropriate aggregation type, which, for the purpose of smooth integration with subsequent processes, should be set to NONE.
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219854578

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 203535 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae, data reload: false
   
   query1	935	434	379	379
   query2	6529	2825	2509	2509
   query3	6921	207	200	200
   query4	21372	18132	18183	18132
   query5	19745	6522	6483	6483
   query6	305	220	226	220
   query7	4164	300	309	300
   query8	427	458	465	458
   query9	3123	2649	2555	2555
   query10	418	324	299	299
   query11	11355	10702	10674	10674
   query12	129	77	77	77
   query13	5606	699	695	695
   query14	17812	13745	13300	13300
   query15	362	253	252	252
   query16	6466	295	266	266
   query17	1698	1439	899	899
   query18	2286	419	414	414
   query19	202	155	148	148
   query20	84	80	80	80
   query21	193	99	98	98
   query22	5125	4988	4987	4987
   query23	32778	31985	32020	31985
   query24	6967	6500	6489	6489
   query25	529	441	434	434
   query26	525	170	167	167
   query27	1874	302	303	302
   query28	6194	2367	2281	2281
   query29	3002	2789	2608	2608
   query30	244	165	170	165
   query31	927	760	767	760
   query32	72	67	62	62
   query33	402	257	267	257
   query34	863	482	490	482
   query35	1116	949	905	905
   query36	1405	1180	1057	1057
   query37	89	63	61	61
   query38	3057	2947	2894	2894
   query39	1392	1333	1347	1333
   query40	207	100	96	96
   query41	48	44	44	44
   query42	85	82	85	82
   query43	773	791	674	674
   query44	1129	732	728	728
   query45	246	238	236	236
   query46	1231	978	953	953
   query47	1774	1790	1789	1789
   query48	1010	731	723	723
   query49	620	387	369	369
   query50	853	599	605	599
   query51	4764	4661	4673	4661
   query52	100	90	86	86
   query53	458	324	340	324
   query54	2664	2470	2475	2470
   query55	86	81	87	81
   query56	246	210	207	207
   query57	1167	1143	1162	1143
   query58	229	204	199	199
   query59	4419	3674	4061	3674
   query60	225	202	219	202
   query61	111	93	95	93
   query62	850	486	439	439
   query63	490	355	348	348
   query64	2454	1512	1535	1512
   query65	3652	3563	3609	3563
   query66	788	373	378	373
   query67	17543	15315	15481	15315
   query68	8439	659	660	659
   query69	576	347	358	347
   query70	1524	1428	1325	1325
   query71	393	316	305	305
   query72	6496	3496	3487	3487
   query73	731	322	322	322
   query74	6275	5918	5897	5897
   query75	4774	3816	3672	3672
   query76	4958	1180	1199	1180
   query77	718	256	248	248
   query78	12512	11682	11870	11682
   query79	9354	651	667	651
   query80	1633	419	407	407
   query81	505	238	233	233
   query82	1685	104	104	104
   query83	166	133	135	133
   query84	265	73	73	73
   query85	892	332	324	324
   query86	343	290	330	290
   query87	3242	3018	3039	3018
   query88	5178	2323	2332	2323
   query89	419	302	351	302
   query90	1927	224	216	216
   query91	177	140	141	140
   query92	61	56	55	55
   query93	6271	579	580	579
   query94	741	215	212	212
   query95	1133	1064	1071	1064
   query96	657	331	328	328
   query97	6509	6296	6443	6296
   query98	205	192	175	175
   query99	2970	838	885	838
   Total cold run time: 316644 ms
   Total hot run time: 203535 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219877864

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae with default session variables
   Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219869590

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.88 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae, data reload: false
   
   query1	0.02	0.02	0.02
   query2	0.06	0.03	0.02
   query3	0.25	0.06	0.05
   query4	1.76	0.07	0.07
   query5	0.54	0.53	0.51
   query6	1.31	0.65	0.62
   query7	0.02	0.01	0.02
   query8	0.03	0.03	0.02
   query9	0.52	0.47	0.48
   query10	0.55	0.54	0.54
   query11	0.12	0.08	0.09
   query12	0.12	0.09	0.09
   query13	0.63	0.61	0.61
   query14	0.80	0.79	0.80
   query15	0.79	0.77	0.76
   query16	0.37	0.35	0.35
   query17	1.01	1.03	1.01
   query18	0.24	0.24	0.24
   query19	1.94	1.86	1.79
   query20	0.02	0.01	0.01
   query21	15.47	0.56	0.55
   query22	1.96	1.99	1.66
   query23	17.47	1.10	0.84
   query24	6.55	1.20	1.33
   query25	0.38	0.11	0.06
   query26	0.77	0.16	0.15
   query27	0.05	0.04	0.03
   query28	6.07	0.84	0.73
   query29	12.61	2.32	2.29
   query30	0.57	0.54	0.53
   query31	2.81	0.40	0.38
   query32	3.38	0.51	0.50
   query33	3.11	3.10	3.06
   query34	15.28	4.77	4.80
   query35	4.86	4.83	4.84
   query36	1.08	1.00	1.01
   query37	0.06	0.05	0.05
   query38	0.03	0.02	0.02
   query39	0.02	0.02	0.01
   query40	0.16	0.14	0.13
   query41	0.06	0.01	0.01
   query42	0.03	0.01	0.01
   query43	0.02	0.01	0.02
   Total cold run time: 103.9 s
   Total hot run time: 30.88 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219793252

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219828946

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 50160 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	20259	4523	4568	4523
   q2	2259	227	160	160
   q3	12786	1967	1957	1957
   q4	10111	1249	1327	1249
   q5	8372	3851	3918	3851
   q6	233	126	129	126
   q7	2052	1636	1599	1599
   q8	9476	2754	2729	2729
   q9	10848	10262	10262	10262
   q10	8673	3526	3543	3526
   q11	408	247	256	247
   q12	476	305	309	305
   q13	18341	3941	4061	3941
   q14	366	331	342	331
   q15	524	455	456	455
   q16	679	577	575	575
   q17	1160	983	971	971
   q18	7322	7058	6934	6934
   q19	1795	1707	1649	1649
   q20	548	292	308	292
   q21	4432	4142	4031	4031
   q22	537	462	447	447
   Total cold run time: 121657 ms
   Total hot run time: 50160 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4372	4302	4295	4295
   q2	329	234	229	229
   q3	4181	4136	4113	4113
   q4	2757	2743	2759	2743
   q5	7153	7115	7053	7053
   q6	238	128	123	123
   q7	3268	2809	2828	2809
   q8	4424	4516	4488	4488
   q9	16926	16781	16829	16781
   q10	4266	4248	4287	4248
   q11	773	695	714	695
   q12	1044	836	860	836
   q13	7211	3737	3755	3737
   q14	451	435	435	435
   q15	525	467	455	455
   q16	738	684	691	684
   q17	3841	3867	3893	3867
   q18	8894	8826	8823	8823
   q19	1744	1753	1673	1673
   q20	2403	2160	2155	2155
   q21	8711	8555	8587	8555
   q22	1026	981	994	981
   Total cold run time: 85275 ms
   Total hot run time: 79778 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219793089

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #37598:
URL: https://github.com/apache/doris/pull/37598


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org