You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2023/11/17 07:46:14 UTC

[PR] [feature](merge-on-write) enable merge-on-write by default [doris]

bobhan1 opened a new pull request, #27188:
URL: https://github.com/apache/doris/pull/27188

   ## Proposed changes
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816208190

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.81 seconds
    stream load tsv:          583 seconds loaded 74807831229 Bytes, about 122 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          27.8 seconds inserted 10000000 Rows, about 359K ops/s
    storage size: 17098680756 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816200671

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816108051

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820498448

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #27188:
URL: https://github.com/apache/doris/pull/27188


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1818227049

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.01 seconds
    stream load tsv:          565 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.1 seconds inserted 10000000 Rows, about 355K ops/s
    storage size: 17099573417 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820795239

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1815885559

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1815902970

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816204124

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816269960

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.67 seconds
    stream load tsv:          581 seconds loaded 74807831229 Bytes, about 122 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.5 seconds inserted 10000000 Rows, about 350K ops/s
    storage size: 17158456047 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820884891

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816080611

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.38 seconds
    stream load tsv:          581 seconds loaded 74807831229 Bytes, about 122 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
    storage size: 17099940833 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820884932

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816156733

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.32 seconds
    stream load tsv:          579 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.5 seconds inserted 10000000 Rows, about 350K ops/s
    storage size: 17099699651 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820696887

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit fc3adfa85180dd376129be3abc3ea0f53c67dbc5, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4890	4640	4649	4640
   q2	359	155	160	155
   q3	2021	1881	1902	1881
   q4	1416	1252	1217	1217
   q5	3915	3904	3987	3904
   q6	241	128	132	128
   q7	1425	888	910	888
   q8	2736	2755	2747	2747
   q9	57773	9561	9671	9561
   q10	12987	3521	3518	3518
   q11	372	241	250	241
   q12	1679	289	290	289
   q13	20478	3802	3784	3784
   q14	315	290	285	285
   q15	588	540	536	536
   q16	664	574	580	574
   q17	1136	991	941	941
   q18	7790	7283	7349	7283
   q19	1717	1688	1666	1666
   q20	555	292	307	292
   q21	7412	3957	3983	3957
   q22	460	366	379	366
   Total cold run time: 130929 ms
   Total hot run time: 48853 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4672	4578	4571	4571
   q2	339	232	278	232
   q3	3988	3967	3969	3967
   q4	2687	2668	2684	2668
   q5	9753	9632	9637	9632
   q6	238	119	120	119
   q7	3002	2455	2474	2455
   q8	4413	4438	4438	4438
   q9	13145	13070	13139	13070
   q10	4104	4180	4205	4180
   q11	778	649	641	641
   q12	971	800	829	800
   q13	4266	3601	3622	3601
   q14	374	332	362	332
   q15	579	525	524	524
   q16	728	691	676	676
   q17	3841	3823	3931	3823
   q18	9586	8973	8989	8973
   q19	1827	1788	1769	1769
   q20	2378	2032	2069	2032
   q21	8826	8466	8384	8384
   q22	885	799	792	792
   Total cold run time: 81380 ms
   Total hot run time: 77679 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816113781

   you can refer to my previous PR: #19017 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816181145

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.41 seconds
    stream load tsv:          578 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17100688470 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816019710

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1818145212

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820446654

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.77 seconds
    stream load tsv:          577 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
    storage size: 17098291592 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #27188:
URL: https://github.com/apache/doris/pull/27188#discussion_r1400553857


##########
docs/zh-CN/docs/data-table/data-model.md:
##########
@@ -416,7 +416,7 @@ mysql> select sum_merge(k2) , group_concat_merge(k3)from aggstate where k1 != 2;
 
 在某些多维分析场景下,用户更关注的是如何保证 Key 的唯一性,即如何获得 Primary Key 唯一性约束。
 因此,我们引入了 Unique 数据模型。在1.2版本之前,该模型本质上是聚合模型的一个特例,也是一种简化的表结构表示方式。
-由于聚合模型的实现方式是读时合并(merge on read),因此在一些聚合查询上性能不佳(参考后续章节[聚合模型的局限性](#聚合模型的局限性)的描述),
+由于聚合模型的实现方式是读时合并(merge on read),因此在一些聚合查询上性能不佳(参考后续章节[聚合模型的局限性](#聚合模型的局限性)的描述),
 在1.2版本我们引入了Unique模型新的实现方式,写时合并(merge on write),通过在写入时做一些额外的工作,实现了最优的查询性能。
 写时合并将在未来替换读时合并成为Unique模型的默认实现方式,两者将会短暂的共存一段时间。下面将对两种实现方式分别举例进行说明。

Review Comment:
   “写时合并将在未来替换读时合并成为Unique模型的默认实现方式,两者将会短暂的共存一段时间”
   update this description



##########
docs/zh-CN/docs/data-table/data-model.md:
##########
@@ -416,7 +416,7 @@ mysql> select sum_merge(k2) , group_concat_merge(k3)from aggstate where k1 != 2;
 
 在某些多维分析场景下,用户更关注的是如何保证 Key 的唯一性,即如何获得 Primary Key 唯一性约束。
 因此,我们引入了 Unique 数据模型。在1.2版本之前,该模型本质上是聚合模型的一个特例,也是一种简化的表结构表示方式。
-由于聚合模型的实现方式是读时合并(merge on read),因此在一些聚合查询上性能不佳(参考后续章节[聚合模型的局限性](#聚合模型的局限性)的描述),
+由于聚合模型的实现方式是读时合并(merge on read),因此在一些聚合查询上性能不佳(参考后续章节[聚合模型的局限性](#聚合模型的局限性)的描述),
 在1.2版本我们引入了Unique模型新的实现方式,写时合并(merge on write),通过在写入时做一些额外的工作,实现了最优的查询性能。
 写时合并将在未来替换读时合并成为Unique模型的默认实现方式,两者将会短暂的共存一段时间。下面将对两种实现方式分别举例进行说明。

Review Comment:
   “写时合并将在未来替换读时合并成为Unique模型的默认实现方式,两者将会短暂的共存一段时间”
   update this description



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1816265186

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.02 seconds
    stream load tsv:          580 seconds loaded 74807831229 Bytes, about 123 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          34 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.6 seconds inserted 10000000 Rows, about 349K ops/s
    storage size: 17099983166 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1815999889

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1818230364

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit db91e487dd3a6749f3962cbb6653459a86aa9f5a, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4958	4687	4655	4655
   q2	360	147	159	147
   q3	2039	1976	1927	1927
   q4	1395	1281	1242	1242
   q5	3951	3943	4033	3943
   q6	256	130	130	130
   q7	1397	878	902	878
   q8	2783	2782	2785	2782
   q9	9899	9509	9343	9343
   q10	3457	3519	3534	3519
   q11	377	242	257	242
   q12	440	300	291	291
   q13	4574	3853	3825	3825
   q14	320	293	282	282
   q15	582	534	518	518
   q16	671	586	589	586
   q17	1132	923	920	920
   q18	8005	7551	7371	7371
   q19	1664	1683	1688	1683
   q20	579	332	325	325
   q21	4447	4035	4008	4008
   q22	483	371	369	369
   Total cold run time: 53769 ms
   Total hot run time: 48986 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4582	4573	4589	4573
   q2	330	223	247	223
   q3	4010	4011	4026	4011
   q4	2742	2725	2726	2725
   q5	9625	9698	9611	9611
   q6	252	121	125	121
   q7	3025	2543	2504	2504
   q8	4395	4417	4408	4408
   q9	12991	12810	12987	12810
   q10	4027	4158	4175	4158
   q11	798	654	691	654
   q12	971	818	824	818
   q13	4297	3553	3534	3534
   q14	391	346	353	346
   q15	554	513	516	513
   q16	750	683	670	670
   q17	3916	3889	3894	3889
   q18	9645	8908	9215	8908
   q19	1777	1784	1774	1774
   q20	2404	2034	2067	2034
   q21	8870	8720	8724	8720
   q22	882	826	812	812
   Total cold run time: 81234 ms
   Total hot run time: 77816 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820379060

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820486464

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit 1bfadb1f21cb6166c4ebc13c6c00c30043f15215, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4913	4646	4652	4646
   q2	369	144	140	140
   q3	2028	1901	1910	1901
   q4	1382	1253	1262	1253
   q5	3929	3944	4018	3944
   q6	240	132	132	132
   q7	1413	897	890	890
   q8	2740	2772	2756	2756
   q9	18425	13553	9666	9666
   q10	13171	3540	3521	3521
   q11	394	254	247	247
   q12	466	293	297	293
   q13	21850	3784	3806	3784
   q14	321	293	291	291
   q15	590	539	527	527
   q16	669	589	567	567
   q17	1151	969	925	925
   q18	7843	7269	7370	7269
   q19	1663	1690	1673	1673
   q20	535	306	308	306
   q21	4435	3935	3937	3935
   q22	464	362	366	362
   Total cold run time: 88991 ms
   Total hot run time: 49028 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4601	4576	4571	4571
   q2	342	235	258	235
   q3	4014	3994	3990	3990
   q4	2707	2696	2713	2696
   q5	9823	9687	9657	9657
   q6	245	122	127	122
   q7	2984	2457	2504	2457
   q8	4440	4424	4397	4397
   q9	13230	13122	13127	13122
   q10	4101	4192	4216	4192
   q11	813	711	648	648
   q12	970	815	831	815
   q13	4275	3598	3595	3595
   q14	389	335	354	335
   q15	572	520	531	520
   q16	733	709	676	676
   q17	3902	3892	3875	3875
   q18	9510	9140	8896	8896
   q19	1768	1782	1796	1782
   q20	2411	2060	2033	2033
   q21	8755	8622	8843	8622
   q22	877	742	807	742
   Total cold run time: 81462 ms
   Total hot run time: 77978 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-on-write) enable merge-on-write by default [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27188:
URL: https://github.com/apache/doris/pull/27188#issuecomment-1820705277

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.89 seconds
    stream load tsv:          573 seconds loaded 74807831229 Bytes, about 124 MB/s
    stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17099339088 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org