You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/02/18 15:23:30 UTC
[GitHub] [incubator-doris] dingjing0604 opened a new issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
dingjing0604 opened a new issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141
### Discussed in https://github.com/apache/incubator-doris/discussions/8130
<div type='discussions-op-text'>
<sup>Originally posted by **dingjing0604** February 18, 2022</sup>
配置 1FE+3BE，高效ESSD盘。测试场景为TPCH100G。
设置并发度为1时，需要22分钟。设置并发度为8、32、64、96时，时间基本都是10分钟左右。
请问这是正常的吗？
为什么别的数据库跑TPCH的时候会在某一并发度取得最佳性能，但是doris的结果都一致？
（以上测试结果都是在同一并发度下跑10次取得的最优结果）</div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] dingjing0604 commented on issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
Posted by GitBox <gi...@apache.org>.
dingjing0604 commented on issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141#issuecomment-1044878820
> 是不是并发度到8的时候实际上CPU 资源已经满了呢？ 另外，能否告知一下 BE的CPU,内存的配置？ 另外，设置并发度用的是哪个参数？
（1）并发度为8、16 、32、64时，3个BE的cpu利用率分别为16% 17% 14%。内存使用率无法从监控中得知。
（2）每个BE的配置为 32核256G，本地盘存储7152G，使用带宽100Mbps；
测试前并未配置cpu和内存的使用限制，be.conf的配置为 ：
storage_root_path = /root/be/data
streaming_load_max_mb=160000
timeout=10000
（3）设置并发度参数使用：set parallel_fragment_exec_instance_num = 8;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141#issuecomment-1045984196
你建表的时候的tablet 数目是多少呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] dingjing0604 commented on issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
Posted by GitBox <gi...@apache.org>.
dingjing0604 commented on issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141#issuecomment-1046056914
> 你建表的时候的tablet 数目是多少呢？
1. 8个表。load之前的数据大小：
-rw-r--r-- 1 root root 2.3G Jan 27 20:16 customer.csv
-rw-r--r-- 1 root root 74G Jan 27 20:30 lineitem.csv
-rw-r--r-- 1 root root 2.2K Jan 27 20:30 nation.csv
-rw-r--r-- 1 root root 17G Jan 27 20:33 orders.csv
-rw-r--r-- 1 root root 2.3G Jan 27 20:35 part.csv
-rw-r--r-- 1 root root 12G Jan 27 20:34 partsupp.csv
-rw-r--r-- 1 root root 384 Jan 27 20:35 region.csv
-rw-r--r-- 1 root root 136M Jan 27 20:35 supplier.csv
2. 建表语句
CREATE TABLE `customer` (
`C_CUSTKEY` int(11) NOT NULL,
`C_NAME` varchar(25) NOT NULL,
`C_ADDRESS` varchar(40) NOT NULL,
`C_NATIONKEY` int(11) NOT NULL,
`C_PHONE` varchar(15) NOT NULL,
`C_ACCTBAL` decimal(12,2) NOT NULL,
`C_MKTSEGMENT` varchar(10) NOT NULL,
`C_COMMENT` varchar(117) NOT NULL
)
DISTRIBUTED BY HASH(C_CUSTKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `lineitem` (
`L_ORDERKEY` bigint(20) NOT NULL,
`L_PARTKEY` int(11) NOT NULL,
`L_SUPPKEY` int(11) NOT NULL,
`L_LINENUMBER` bigint(20) NOT NULL,
`L_QUANTITY` decimal(12,2) NOT NULL,
`L_EXTENDEDPRICE` decimal(12,2) NOT NULL,
`L_DISCOUNT` decimal(12,2) NOT NULL,
`L_TAX` decimal(12,2) NOT NULL,
`L_RETURNFLAG` varchar(1) NOT NULL,
`L_LINESTATUS` varchar(1) NOT NULL,
`L_SHIPDATE` date NOT NULL,
`L_COMMITDATE` date NOT NULL,
`L_RECEIPTDATE` date NOT NULL,
`L_SHIPINSTRUCT` varchar(25) NOT NULL,
`L_SHIPMODE` varchar(10) NOT NULL,
`L_COMMENT` varchar(44) NOT NULL
)
DISTRIBUTED BY HASH(L_ORDERKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `nation` (
`N_NATIONKEY` int(11) NOT NULL,
`N_NAME` varchar(25) NOT NULL,
`N_REGIONKEY` int(11) NOT NULL,
`N_COMMENT` varchar(152) DEFAULT NULL
)
DISTRIBUTED BY HASH(N_NATIONKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `orders` (
`O_ORDERKEY` bigint(20) NOT NULL,
`O_CUSTKEY` int(11) NOT NULL,
`O_ORDERSTATUS` varchar(1) NOT NULL,
`O_TOTALPRICE` decimal(12,2) NOT NULL,
`O_ORDERDATE` date NOT NULL,
`O_ORDERPRIORITY` varchar(15) NOT NULL,
`O_CLERK` varchar(15) NOT NULL,
`O_SHIPPRIORITY` int(11) NOT NULL,
`O_COMMENT` varchar(79) NOT NULL
)
DISTRIBUTED BY HASH(O_ORDERKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `part` (
`P_PARTKEY` int(11) NOT NULL,
`P_NAME` varchar(55) NOT NULL,
`P_MFGR` varchar(25) NOT NULL,
`P_BRAND` varchar(10) NOT NULL,
`P_TYPE` varchar(25) NOT NULL,
`P_SIZE` int(11) NOT NULL,
`P_CONTAINER` varchar(10) NOT NULL,
`P_RETAILPRICE` decimal(12,2) NOT NULL,
`P_COMMENT` varchar(23) NOT NULL
)
DISTRIBUTED BY HASH(P_PARTKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `partsupp` (
`PS_PARTKEY` int(11) NOT NULL,
`PS_SUPPKEY` int(11) NOT NULL,
`PS_AVAILQTY` int(11) NOT NULL,
`PS_SUPPLYCOST` decimal(12,2) NOT NULL,
`PS_COMMENT` varchar(199) NOT NULL
)
DISTRIBUTED BY HASH(PS_PARTKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `region` (
`R_REGIONKEY` int(11) NOT NULL,
`R_NAME` varchar(25) NOT NULL,
`R_COMMENT` varchar(152) DEFAULT NULL
)
DISTRIBUTED BY HASH(R_REGIONKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
CREATE TABLE `supplier` (
`S_SUPPKEY` int(11) NOT NULL,
`S_NAME` varchar(25) NOT NULL,
`S_ADDRESS` varchar(40) NOT NULL,
`S_NATIONKEY` int(11) NOT NULL,
`S_PHONE` varchar(15) NOT NULL,
`S_ACCTBAL` decimal(12,2) NOT NULL,
`S_COMMENT` varchar(101) NOT NULL
)
DISTRIBUTED BY HASH(S_SUPPKEY) BUCKETS 10
PROPERTIES("replication_num" = "1");
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141#issuecomment-1044703921
是不是并发度到8的时候实际上CPU 资源已经满了呢？
另外，能否告知一下 BE的CPU,内存的配置？
另外，设置并发度用的是哪个参数？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141#issuecomment-1046059715
你把bucket 数目调整成64试试
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Userwhite commented on issue #8141: doris跑tpch并发设置8、16、32、64、96，结果都是10分钟左右
Posted by GitBox <gi...@apache.org>.
Userwhite commented on issue #8141:
URL: https://github.com/apache/incubator-doris/issues/8141#issuecomment-1047435002
10tablet限制了最高并发度就是10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org