You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/20 03:53:12 UTC
[GitHub] [incubator-doris] hello-stephen opened a new issue, #9698: [Enhancement] table data is unevenly distributed on 3 BEs
hello-stephen opened a new issue, #9698:
URL: https://github.com/apache/incubator-doris/issues/9698
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
I use scripts from tools/ssb-tools to load ssb data of scale 100 into database ssb100, set 1 replica in create table sql, after it finished, I found that data is unevenly distributed, which may affect the performance.
During the ssb loading, tpch data is also loading, I'm not sure whether it has any effect.
After all loading task finished, I run ssb load script again to load same data into database ssb100_1r, this time, data is evenly distributed.
So, the data distribution algorithm may have some space to become more stable.
```
mysql> admin show replica distribution from ssb100.lineorder;
+-----------+------------+-------------+------------------------------------------------+------------+------------------------------------------------+-------------+
| BackendId | ReplicaNum | ReplicaSize | NumGraph | NumPercent | SizeGraph | SizePercent |
+-----------+------------+-------------+------------------------------------------------+------------+------------------------------------------------+-------------+
| 10002 | 157 | 7392936424 | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 46.73 % | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 46.38 % |
| 10003 | 141 | 6735012816 | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 41.96 % | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 42.25 % |
| 10004 | 38 | 1813474345 | >>>>>>>>>>> | 11.31 % | >>>>>>>>>>> | 11.38 % |
+-----------+------------+-------------+------------------------------------------------+------------+------------------------------------------------+-------------+
3 rows in set (0.01 sec)
mysql> admin show replica distribution from ssb100_1r.lineorder;
+-----------+------------+-------------+------------------------------------+------------+------------------------------------+-------------+
| BackendId | ReplicaNum | ReplicaSize | NumGraph | NumPercent | SizeGraph | SizePercent |
+-----------+------------+-------------+------------------------------------+------------+------------------------------------+-------------+
| 10002 | 117 | 3339913152 | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 34.82 % | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 34.92 % |
| 10003 | 109 | 3109652757 | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 32.44 % | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 32.51 % |
| 10004 | 110 | 3114643318 | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 32.74 % | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 32.57 % |
+-----------+------------+-------------+------------------------------------+------------+------------------------------------+-------------+
3 rows in set (0.00 sec)
mysql> show create table ssb100.lineorder;
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table
|
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lineorder | CREATE TABLE `lineorder` (
`lo_orderkey` bigint(20) NOT NULL COMMENT "",
`lo_linenumber` bigint(20) NOT NULL COMMENT "",
`lo_custkey` int(11) NOT NULL COMMENT "",
`lo_partkey` int(11) NOT NULL COMMENT "",
`lo_suppkey` int(11) NOT NULL COMMENT "",
`lo_orderdate` int(11) NOT NULL COMMENT "",
`lo_orderpriority` varchar(16) NOT NULL COMMENT "",
`lo_shippriority` int(11) NOT NULL COMMENT "",
`lo_quantity` bigint(20) NOT NULL COMMENT "",
`lo_extendedprice` bigint(20) NOT NULL COMMENT "",
`lo_ordtotalprice` bigint(20) NOT NULL COMMENT "",
`lo_discount` bigint(20) NOT NULL COMMENT "",
`lo_revenue` bigint(20) NOT NULL COMMENT "",
`lo_supplycost` bigint(20) NOT NULL COMMENT "",
`lo_tax` bigint(20) NOT NULL COMMENT "",
`lo_commitdate` bigint(20) NOT NULL COMMENT "",
`lo_shipmode` varchar(11) NOT NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`lo_orderkey`, `lo_linenumber`, `lo_custkey`)
COMMENT "OLAP"
PARTITION BY RANGE(`lo_orderdate`)
(PARTITION p1992 VALUES [("-2147483648"), ("19930101")),
PARTITION p1993 VALUES [("19930101"), ("19940101")),
PARTITION p1994 VALUES [("19940101"), ("19950101")),
PARTITION p1995 VALUES [("19950101"), ("19960101")),
PARTITION p1996 VALUES [("19960101"), ("19970101")),
PARTITION p1997 VALUES [("19970101"), ("19980101")),
PARTITION p1998 VALUES [("19980101"), ("19990101")))
DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 48
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
) |
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
```
### Solution
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org