You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/12/01 08:00:55 UTC
[GitHub] [incubator-doris] qidaye opened a new issue #4992: [bug] finished load jobs cost too much heap memory
qidaye opened a new issue #4992:
URL: https://github.com/apache/incubator-doris/issues/4992
**Describe the bug**
Current, labels of finished or cancelled load jobs will be removed after *label_keep_max_second*, default 3 days. All load jobs' info is kept in memory before being removed.
Load job contents are as follows:
**Note**:
TTabletLocation_count = partition_count * index_count * tablet_count
TOlapTablePartition_count = partition_count
TOlapTableIndexTablets_count = partition_count * index_count
When load job is **cancelled**, the load job tasks will be cleared.
When load job is **finished**, the load job tasks are still being cached.
This is a huge heap memory cost, when load job is kept too many and table has many partitions and buckets. This ultimately leads to Doris FE unavailable.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a table with many partitions and tablets
```sql
CREATE TABLE `aoi_pv` (
`date` date NULL COMMENT "日期",
`bow_id` int(11) NULL COMMENT "词包Id",
`aoi_id` bigint(20) NULL COMMENT "小区Id",
`city_id` smallint(6) NULL COMMENT "城市Id",
`camp_id` int(11) NULL COMMENT "活动Id",
`is_sel` tinyint(4) REPLACE NULL COMMENT "是否Sel",
`pv` bigint(20) REPLACE NULL COMMENT "pv",
`city_pv` bigint(20) REPLACE NULL COMMENT "pv"
) ENGINE=OLAP
UNIQUE KEY(`date`, `bow_id`, `aoi_id`, `city_id`, `camp_id`)
COMMENT "OLAP"
PARTITION BY RANGE(`date`)
(PARTITION p20190501 VALUES [('1900-01-01'), ('2019-05-01')),
PARTITION p20190731 VALUES [('2019-05-01'), ('2019-08-01')),
PARTITION p20190801 VALUES [('2019-08-01'), ('2019-08-02')),
PARTITION p20190802 VALUES [('2019-08-02'), ('2019-08-03')),
PARTITION p20190803 VALUES [('2019-08-03'), ('2019-08-04')),
PARTITION p20190804 VALUES [('2019-08-04'), ('2019-08-05')),
PARTITION p20190805 VALUES [('2019-08-05'), ('2019-08-06')),
PARTITION p20190806 VALUES [('2019-08-06'), ('2019-08-07')),
PARTITION p20190807 VALUES [('2019-08-07'), ('2019-08-08')),
PARTITION p20190808 VALUES [('2019-08-08'), ('2019-08-09')),
PARTITION p20190809 VALUES [('2019-08-09'), ('2019-08-10')),
PARTITION p20190810 VALUES [('2019-08-10'), ('2019-08-11')),
PARTITION p20190811 VALUES [('2019-08-11'), ('2019-08-12')),
PARTITION p20190812 VALUES [('2019-08-12'), ('2019-08-13')),
PARTITION p20190813 VALUES [('2019-08-13'), ('2019-08-14')),
PARTITION p20190814 VALUES [('2019-08-14'), ('2019-08-15')),
PARTITION p20190815 VALUES [('2019-08-15'), ('2019-08-16')),
PARTITION p20190816 VALUES [('2019-08-16'), ('2019-08-17')),
PARTITION p20190817 VALUES [('2019-08-17'), ('2019-08-18')),
PARTITION p20190818 VALUES [('2019-08-18'), ('2019-08-19')),
PARTITION p20190819 VALUES [('2019-08-19'), ('2019-08-20')),
PARTITION p20190820 VALUES [('2019-08-20'), ('2019-08-21')),
PARTITION p20190821 VALUES [('2019-08-21'), ('2019-08-22')),
PARTITION p20190822 VALUES [('2019-08-22'), ('2019-08-23')),
PARTITION p20190823 VALUES [('2019-08-23'), ('2019-08-24')),
PARTITION p20190824 VALUES [('2019-08-24'), ('2019-08-25')),
PARTITION p20190825 VALUES [('2019-08-25'), ('2019-08-26')),
PARTITION p20190826 VALUES [('2019-08-26'), ('2019-08-27')),
PARTITION p20190827 VALUES [('2019-08-27'), ('2019-08-28')),
PARTITION p20190828 VALUES [('2019-08-28'), ('2019-08-29')),
PARTITION p20190829 VALUES [('2019-08-29'), ('2019-08-30')),
PARTITION p20190830 VALUES [('2019-08-30'), ('2019-08-31')),
PARTITION p20190831 VALUES [('2019-08-31'), ('2019-09-01')),
PARTITION p20190901 VALUES [('2019-09-01'), ('2019-09-02')),
PARTITION p20190902 VALUES [('2019-09-02'), ('2019-09-03')),
PARTITION p20190903 VALUES [('2019-09-03'), ('2019-09-04')),
PARTITION p20190904 VALUES [('2019-09-04'), ('2019-09-05')),
PARTITION p20190905 VALUES [('2019-09-05'), ('2019-09-06')),
PARTITION p20190906 VALUES [('2019-09-06'), ('2019-09-07')),
PARTITION p20190907 VALUES [('2019-09-07'), ('2019-09-08')),
PARTITION p20190908 VALUES [('2019-09-08'), ('2019-09-09')),
PARTITION p20190909 VALUES [('2019-09-09'), ('2019-09-10')),
PARTITION p20190910 VALUES [('2019-09-10'), ('2019-09-11')),
PARTITION p20190911 VALUES [('2019-09-11'), ('2019-09-12')),
PARTITION p20190912 VALUES [('2019-09-12'), ('2019-09-13')),
PARTITION p20190913 VALUES [('2019-09-13'), ('2019-09-14')),
PARTITION p20190914 VALUES [('2019-09-14'), ('2019-09-15')),
PARTITION p20190915 VALUES [('2019-09-15'), ('2019-09-16')),
PARTITION p20190916 VALUES [('2019-09-16'), ('2019-09-17')),
PARTITION p20190917 VALUES [('2019-09-17'), ('2019-09-18')),
PARTITION p20190918 VALUES [('2019-09-18'), ('2019-09-19')),
PARTITION p20190919 VALUES [('2019-09-19'), ('2019-09-20')),
PARTITION p20190920 VALUES [('2019-09-20'), ('2019-09-21')),
PARTITION p20190921 VALUES [('2019-09-21'), ('2019-09-22')),
PARTITION p20190922 VALUES [('2019-09-22'), ('2019-09-23')),
PARTITION p20190923 VALUES [('2019-09-23'), ('2019-09-24')),
PARTITION p20190924 VALUES [('2019-09-24'), ('2019-09-25')),
PARTITION p20190925 VALUES [('2019-09-25'), ('2019-09-26')),
PARTITION p20190926 VALUES [('2019-09-26'), ('2019-09-27')),
PARTITION p20190927 VALUES [('2019-09-27'), ('2019-09-28')),
PARTITION p20190928 VALUES [('2019-09-28'), ('2019-09-29')),
PARTITION p20190929 VALUES [('2019-09-29'), ('2019-09-30')),
PARTITION p20190930 VALUES [('2019-09-30'), ('2019-10-01')),
PARTITION p20191001 VALUES [('2019-10-01'), ('2019-10-02')),
PARTITION p20191002 VALUES [('2019-10-02'), ('2019-10-03')),
PARTITION p20191003 VALUES [('2019-10-03'), ('2019-10-04')),
PARTITION p20191004 VALUES [('2019-10-04'), ('2019-10-05')),
PARTITION p20191005 VALUES [('2019-10-05'), ('2019-10-06')),
PARTITION p20191006 VALUES [('2019-10-06'), ('2019-10-07')),
PARTITION p20191007 VALUES [('2019-10-07'), ('2019-10-08')),
PARTITION p20191008 VALUES [('2019-10-08'), ('2019-10-09')),
PARTITION p20191009 VALUES [('2019-10-09'), ('2019-10-10')),
PARTITION p20191010 VALUES [('2019-10-10'), ('2019-10-11')),
PARTITION p20191011 VALUES [('2019-10-11'), ('2019-10-12')),
PARTITION p20191012 VALUES [('2019-10-12'), ('2019-10-13')),
PARTITION p20191013 VALUES [('2019-10-13'), ('2019-10-14')),
PARTITION p20191014 VALUES [('2019-10-14'), ('2019-10-15')),
PARTITION p20191015 VALUES [('2019-10-15'), ('2019-10-16')),
PARTITION p20191016 VALUES [('2019-10-16'), ('2019-10-17')),
PARTITION p20191017 VALUES [('2019-10-17'), ('2019-10-18')),
PARTITION p20191018 VALUES [('2019-10-18'), ('2019-10-19')),
PARTITION p20191019 VALUES [('2019-10-19'), ('2019-10-20')),
PARTITION p20191020 VALUES [('2019-10-20'), ('2019-10-21')),
PARTITION p20191021 VALUES [('2019-10-21'), ('2019-10-22')),
PARTITION p20191022 VALUES [('2019-10-22'), ('2019-10-23')),
PARTITION p20191023 VALUES [('2019-10-23'), ('2019-10-24')),
PARTITION p20191024 VALUES [('2019-10-24'), ('2019-10-25')),
PARTITION p20191025 VALUES [('2019-10-25'), ('2019-10-26')),
PARTITION p20191026 VALUES [('2019-10-26'), ('2019-10-27')),
PARTITION p20191027 VALUES [('2019-10-27'), ('2019-10-28')),
PARTITION p20191028 VALUES [('2019-10-28'), ('2019-10-29')),
PARTITION p20191029 VALUES [('2019-10-29'), ('2019-10-30')),
PARTITION p20191030 VALUES [('2019-10-30'), ('2019-10-31')),
PARTITION p20191031 VALUES [('2019-10-31'), ('2019-11-01')),
PARTITION p20191101 VALUES [('2019-11-01'), ('2019-11-02')),
PARTITION p20191102 VALUES [('2019-11-02'), ('2019-11-03')),
PARTITION p20191103 VALUES [('2019-11-03'), ('2019-11-04')),
PARTITION p20191104 VALUES [('2019-11-04'), ('2019-11-05')),
PARTITION p20191105 VALUES [('2019-11-05'), ('2019-11-06')),
PARTITION p20191106 VALUES [('2019-11-06'), ('2019-11-07')),
PARTITION p20191107 VALUES [('2019-11-07'), ('2019-11-08')),
PARTITION p20191108 VALUES [('2019-11-08'), ('2019-11-09')),
PARTITION p20191109 VALUES [('2019-11-09'), ('2019-11-10')),
PARTITION p20191110 VALUES [('2019-11-10'), ('2019-11-11')),
PARTITION p20191111 VALUES [('2019-11-11'), ('2019-11-12')),
PARTITION p20191112 VALUES [('2019-11-12'), ('2019-11-13')),
PARTITION p20191113 VALUES [('2019-11-13'), ('2019-11-14')),
PARTITION p20191114 VALUES [('2019-11-14'), ('2019-11-15')),
PARTITION p20191115 VALUES [('2019-11-15'), ('2019-11-16')),
PARTITION p20191116 VALUES [('2019-11-16'), ('2019-11-17')),
PARTITION p20191117 VALUES [('2019-11-17'), ('2019-11-18')),
PARTITION p20191118 VALUES [('2019-11-18'), ('2019-11-19')),
PARTITION p20191120 VALUES [('2019-11-19'), ('2019-11-20')),
PARTITION p20191121 VALUES [('2019-11-20'), ('2019-11-21')),
PARTITION p20191122 VALUES [('2019-11-21'), ('2019-11-22')),
PARTITION p20191123 VALUES [('2019-11-22'), ('2019-11-23')),
PARTITION p20191124 VALUES [('2019-11-23'), ('2019-11-24')),
PARTITION p20191125 VALUES [('2019-11-24'), ('2019-11-25')),
PARTITION p20191126 VALUES [('2019-11-25'), ('2019-11-26')),
PARTITION p20191127 VALUES [('2019-11-26'), ('2019-11-27')),
PARTITION p20191128 VALUES [('2019-11-27'), ('2019-11-28')),
PARTITION p20191129 VALUES [('2019-11-28'), ('2019-11-29')),
PARTITION p20191130 VALUES [('2019-11-29'), ('2019-11-30')),
PARTITION p20191201 VALUES [('2019-11-30'), ('2019-12-01')),
PARTITION p20191202 VALUES [('2019-12-01'), ('2019-12-02')),
PARTITION p20191203 VALUES [('2019-12-02'), ('2019-12-03')),
PARTITION p20191204 VALUES [('2019-12-03'), ('2019-12-04')),
PARTITION p20191205 VALUES [('2019-12-04'), ('2019-12-05')),
PARTITION p20191206 VALUES [('2019-12-05'), ('2019-12-06')),
PARTITION p20191207 VALUES [('2019-12-06'), ('2019-12-07')),
PARTITION p20191208 VALUES [('2019-12-07'), ('2019-12-08')),
PARTITION p20191209 VALUES [('2019-12-08'), ('2019-12-09')),
PARTITION p20191210 VALUES [('2019-12-09'), ('2019-12-10')),
PARTITION p20191211 VALUES [('2019-12-10'), ('2019-12-11')),
PARTITION p20191212 VALUES [('2019-12-11'), ('2019-12-12')),
PARTITION p20191213 VALUES [('2019-12-12'), ('2019-12-13')),
PARTITION p20191214 VALUES [('2019-12-13'), ('2019-12-14')),
PARTITION p20191215 VALUES [('2019-12-14'), ('2019-12-15')),
PARTITION p20191216 VALUES [('2019-12-15'), ('2019-12-16')),
PARTITION p20191217 VALUES [('2019-12-16'), ('2019-12-17')),
PARTITION p20191218 VALUES [('2019-12-17'), ('2019-12-18')),
PARTITION p20191219 VALUES [('2019-12-18'), ('2019-12-19')),
PARTITION p20191220 VALUES [('2019-12-19'), ('2019-12-20')),
PARTITION p20191221 VALUES [('2019-12-20'), ('2019-12-21')),
PARTITION p20191222 VALUES [('2019-12-21'), ('2019-12-22')),
PARTITION p20191223 VALUES [('2019-12-22'), ('2019-12-23')),
PARTITION p20191224 VALUES [('2019-12-23'), ('2019-12-24')),
PARTITION p20191225 VALUES [('2019-12-24'), ('2019-12-25')),
PARTITION p20191226 VALUES [('2019-12-25'), ('2019-12-26')),
PARTITION p20191227 VALUES [('2019-12-26'), ('2019-12-27')),
PARTITION p20191228 VALUES [('2019-12-27'), ('2019-12-28')),
PARTITION p20191229 VALUES [('2019-12-28'), ('2019-12-29')),
PARTITION p20191230 VALUES [('2019-12-29'), ('2019-12-30')),
PARTITION p20191231 VALUES [('2019-12-30'), ('2019-12-31')),
PARTITION p20200101 VALUES [('2019-12-31'), ('2020-01-02')),
PARTITION p20200102 VALUES [('2020-01-02'), ('2020-01-03')),
PARTITION p20200103 VALUES [('2020-01-03'), ('2020-01-04')),
PARTITION p20200104 VALUES [('2020-01-04'), ('2020-01-05')),
PARTITION p20200105 VALUES [('2020-01-05'), ('2020-01-06')),
PARTITION p20200106 VALUES [('2020-01-06'), ('2020-01-07')),
PARTITION p20200107 VALUES [('2020-01-07'), ('2020-01-08')),
PARTITION p20200108 VALUES [('2020-01-08'), ('2020-01-09')),
PARTITION p20200109 VALUES [('2020-01-09'), ('2020-01-10')),
PARTITION p20200110 VALUES [('2020-01-10'), ('2020-01-11')),
PARTITION p20200111 VALUES [('2020-01-11'), ('2020-01-12')),
PARTITION p20200112 VALUES [('2020-01-12'), ('2020-01-13')),
PARTITION p20200113 VALUES [('2020-01-13'), ('2020-01-14')),
PARTITION p20200114 VALUES [('2020-01-14'), ('2020-01-15')),
PARTITION p20200115 VALUES [('2020-01-15'), ('2020-01-16')),
PARTITION p20200116 VALUES [('2020-01-16'), ('2020-01-17')),
PARTITION p20200117 VALUES [('2020-01-17'), ('2020-01-18')),
PARTITION p20200118 VALUES [('2020-01-18'), ('2020-01-19')),
PARTITION p20200119 VALUES [('2020-01-19'), ('2020-01-20')),
PARTITION p20200120 VALUES [('2020-01-20'), ('2020-01-21')),
PARTITION p20200121 VALUES [('2020-01-21'), ('2020-01-22')),
PARTITION p20200122 VALUES [('2020-01-22'), ('2020-01-23')),
PARTITION p20200123 VALUES [('2020-01-23'), ('2020-01-24')),
PARTITION p20200124 VALUES [('2020-01-24'), ('2020-01-25')),
PARTITION p20200125 VALUES [('2020-01-25'), ('2020-01-26')),
PARTITION p20200126 VALUES [('2020-01-26'), ('2020-01-27')),
PARTITION p20200127 VALUES [('2020-01-27'), ('2020-01-28')),
PARTITION p20200128 VALUES [('2020-01-28'), ('2020-01-29')),
PARTITION p20200129 VALUES [('2020-01-29'), ('2020-01-30')),
PARTITION p20200130 VALUES [('2020-01-30'), ('2020-01-31')),
PARTITION p20200131 VALUES [('2020-01-31'), ('2020-02-01')),
PARTITION p20200201 VALUES [('2020-02-01'), ('2020-02-02')),
PARTITION p20200202 VALUES [('2020-02-02'), ('2020-02-03')),
PARTITION p20200203 VALUES [('2020-02-03'), ('2020-02-04')),
PARTITION p20200204 VALUES [('2020-02-04'), ('2020-02-05')),
PARTITION p20200205 VALUES [('2020-02-05'), ('2020-02-06')),
PARTITION p20200206 VALUES [('2020-02-06'), ('2020-02-07')),
PARTITION p20200207 VALUES [('2020-02-07'), ('2020-02-08')),
PARTITION p20200208 VALUES [('2020-02-08'), ('2020-02-09')),
PARTITION p20200209 VALUES [('2020-02-09'), ('2020-02-10')),
PARTITION p20200210 VALUES [('2020-02-10'), ('2020-02-11')),
PARTITION p20200211 VALUES [('2020-02-11'), ('2020-02-12')),
PARTITION p20200212 VALUES [('2020-02-12'), ('2020-02-13')),
PARTITION p20200213 VALUES [('2020-02-13'), ('2020-02-14')),
PARTITION p20200214 VALUES [('2020-02-14'), ('2020-02-15')),
PARTITION p20200215 VALUES [('2020-02-15'), ('2020-02-16')),
PARTITION p20200216 VALUES [('2020-02-16'), ('2020-02-17')),
PARTITION p20200217 VALUES [('2020-02-17'), ('2020-02-18')),
PARTITION p20200218 VALUES [('2020-02-18'), ('2020-02-19')),
PARTITION p20200219 VALUES [('2020-02-19'), ('2020-02-20')),
PARTITION p20200220 VALUES [('2020-02-20'), ('2020-02-21')),
PARTITION p20200221 VALUES [('2020-02-21'), ('2020-02-22')),
PARTITION p20200222 VALUES [('2020-02-22'), ('2020-02-23')),
PARTITION p20200223 VALUES [('2020-02-23'), ('2020-02-24')),
PARTITION p20200224 VALUES [('2020-02-24'), ('2020-02-25')),
PARTITION p20200225 VALUES [('2020-02-25'), ('2020-02-26')),
PARTITION p20200226 VALUES [('2020-02-26'), ('2020-02-27')),
PARTITION p20200227 VALUES [('2020-02-27'), ('2020-02-28')),
PARTITION p20200228 VALUES [('2020-02-28'), ('2020-02-29')),
PARTITION p20200229 VALUES [('2020-02-29'), ('2020-03-01')),
PARTITION p20200301 VALUES [('2020-03-01'), ('2020-03-02')),
PARTITION p20200302 VALUES [('2020-03-02'), ('2020-03-03')),
PARTITION p20200303 VALUES [('2020-03-03'), ('2020-03-04')),
PARTITION p20200304 VALUES [('2020-03-04'), ('2020-03-05')),
PARTITION p20200305 VALUES [('2020-03-05'), ('2020-03-06')),
PARTITION p20200306 VALUES [('2020-03-06'), ('2020-03-07')),
PARTITION p20200307 VALUES [('2020-03-07'), ('2020-03-08')),
PARTITION p20200308 VALUES [('2020-03-08'), ('2020-03-09')),
PARTITION p20200309 VALUES [('2020-03-09'), ('2020-03-10')),
PARTITION p20200310 VALUES [('2020-03-10'), ('2020-03-11')),
PARTITION p20200311 VALUES [('2020-03-11'), ('2020-03-12')),
PARTITION p20200312 VALUES [('2020-03-12'), ('2020-03-13')),
PARTITION p20200313 VALUES [('2020-03-13'), ('2020-03-14')),
PARTITION p20200314 VALUES [('2020-03-14'), ('2020-03-15')),
PARTITION p20200315 VALUES [('2020-03-15'), ('2020-03-16')),
PARTITION p20200316 VALUES [('2020-03-16'), ('2020-03-17')),
PARTITION p20200317 VALUES [('2020-03-17'), ('2020-03-18')),
PARTITION p20200318 VALUES [('2020-03-18'), ('2020-03-19')),
PARTITION p20200319 VALUES [('2020-03-19'), ('2020-03-20')),
PARTITION p20200320 VALUES [('2020-03-20'), ('2020-03-21')),
PARTITION p20200321 VALUES [('2020-03-21'), ('2020-03-22')),
PARTITION p20200322 VALUES [('2020-03-22'), ('2020-03-23')),
PARTITION p20200323 VALUES [('2020-03-23'), ('2020-03-24')),
PARTITION p20200324 VALUES [('2020-03-24'), ('2020-03-25')),
PARTITION p20200325 VALUES [('2020-03-25'), ('2020-03-26')),
PARTITION p20200326 VALUES [('2020-03-26'), ('2020-03-27')),
PARTITION p20200327 VALUES [('2020-03-27'), ('2020-03-28')),
PARTITION p20200328 VALUES [('2020-03-28'), ('2020-03-29')),
PARTITION p20200329 VALUES [('2020-03-29'), ('2020-03-30')),
PARTITION p20200330 VALUES [('2020-03-30'), ('2020-03-31')),
PARTITION p20200331 VALUES [('2020-03-31'), ('2020-04-01')),
PARTITION p20200401 VALUES [('2020-04-01'), ('2020-04-02')),
PARTITION p20200402 VALUES [('2020-04-02'), ('2020-04-03')),
PARTITION p20200403 VALUES [('2020-04-03'), ('2020-04-04')),
PARTITION p20200404 VALUES [('2020-04-04'), ('2020-04-05')),
PARTITION p20200405 VALUES [('2020-04-05'), ('2020-04-06')),
PARTITION p20200406 VALUES [('2020-04-06'), ('2020-04-07')),
PARTITION p20200407 VALUES [('2020-04-07'), ('2020-04-08')),
PARTITION p20200408 VALUES [('2020-04-08'), ('2020-04-09')),
PARTITION p20200409 VALUES [('2020-04-09'), ('2020-04-10')),
PARTITION p20200410 VALUES [('2020-04-10'), ('2020-04-11')),
PARTITION p20200411 VALUES [('2020-04-11'), ('2020-04-12')),
PARTITION p20200412 VALUES [('2020-04-12'), ('2020-04-13')),
PARTITION p20200413 VALUES [('2020-04-13'), ('2020-04-14')),
PARTITION p20200414 VALUES [('2020-04-14'), ('2020-04-15')),
PARTITION p20200415 VALUES [('2020-04-15'), ('2020-04-16')),
PARTITION p20200416 VALUES [('2020-04-16'), ('2020-04-17')),
PARTITION p20200417 VALUES [('2020-04-17'), ('2020-04-18')),
PARTITION p20200418 VALUES [('2020-04-18'), ('2020-04-19')),
PARTITION p20200419 VALUES [('2020-04-19'), ('2020-04-20')),
PARTITION p20200420 VALUES [('2020-04-20'), ('2020-04-21')),
PARTITION p20200421 VALUES [('2020-04-21'), ('2020-04-22')),
PARTITION p20200422 VALUES [('2020-04-22'), ('2020-04-23')),
PARTITION p20200423 VALUES [('2020-04-23'), ('2020-04-24')),
PARTITION p20200424 VALUES [('2020-04-24'), ('2020-04-25')),
PARTITION p20200425 VALUES [('2020-04-25'), ('2020-04-26')),
PARTITION p20200426 VALUES [('2020-04-26'), ('2020-04-27')),
PARTITION p20200427 VALUES [('2020-04-27'), ('2020-04-28')),
PARTITION p20200428 VALUES [('2020-04-28'), ('2020-04-29')),
PARTITION p20200429 VALUES [('2020-04-29'), ('2020-04-30')),
PARTITION p20200430 VALUES [('2020-04-30'), ('2020-05-01')),
PARTITION p20200501 VALUES [('2020-05-01'), ('2020-05-02')),
PARTITION p20200502 VALUES [('2020-05-02'), ('2020-05-03')),
PARTITION p20200503 VALUES [('2020-05-03'), ('2020-05-04')),
PARTITION p20200504 VALUES [('2020-05-04'), ('2020-05-05')),
PARTITION p20200505 VALUES [('2020-05-05'), ('2020-05-06')),
PARTITION p20200506 VALUES [('2020-05-06'), ('2020-05-07')),
PARTITION p20200507 VALUES [('2020-05-07'), ('2020-05-08')),
PARTITION p20200508 VALUES [('2020-05-08'), ('2020-05-09')),
PARTITION p20200509 VALUES [('2020-05-09'), ('2020-05-10')),
PARTITION p20200510 VALUES [('2020-05-10'), ('2020-05-11')),
PARTITION p20200511 VALUES [('2020-05-11'), ('2020-05-12')),
PARTITION p20200512 VALUES [('2020-05-12'), ('2020-05-13')),
PARTITION p20200513 VALUES [('2020-05-13'), ('2020-05-14')),
PARTITION p20200514 VALUES [('2020-05-14'), ('2020-05-15')),
PARTITION p20200515 VALUES [('2020-05-15'), ('2020-05-16')),
PARTITION p20200516 VALUES [('2020-05-16'), ('2020-05-17')),
PARTITION p20200517 VALUES [('2020-05-17'), ('2020-05-18')),
PARTITION p20200518 VALUES [('2020-05-18'), ('2020-05-19')),
PARTITION p20200519 VALUES [('2020-05-19'), ('2020-05-20')),
PARTITION p20200520 VALUES [('2020-05-20'), ('2020-05-21')),
PARTITION p20200521 VALUES [('2020-05-21'), ('2020-05-22')),
PARTITION p20200522 VALUES [('2020-05-22'), ('2020-05-23')),
PARTITION p20200523 VALUES [('2020-05-23'), ('2020-05-24')),
PARTITION p20200524 VALUES [('2020-05-24'), ('2020-05-25')),
PARTITION p20200525 VALUES [('2020-05-25'), ('2020-05-26')),
PARTITION p20200526 VALUES [('2020-05-26'), ('2020-05-27')),
PARTITION p20200527 VALUES [('2020-05-27'), ('2020-05-28')),
PARTITION p20200528 VALUES [('2020-05-28'), ('2020-05-29')),
PARTITION p20200529 VALUES [('2020-05-29'), ('2020-05-30')),
PARTITION p20200530 VALUES [('2020-05-30'), ('2020-05-31')),
PARTITION p20200531 VALUES [('2020-05-31'), ('2020-06-01')),
PARTITION p20200601 VALUES [('2020-06-01'), ('2020-06-02')),
PARTITION p20200602 VALUES [('2020-06-02'), ('2020-06-03')),
PARTITION p20200603 VALUES [('2020-06-03'), ('2020-06-04')),
PARTITION p20200604 VALUES [('2020-06-04'), ('2020-06-05')),
PARTITION p20200605 VALUES [('2020-06-05'), ('2020-06-06')),
PARTITION p20200606 VALUES [('2020-06-06'), ('2020-06-07')),
PARTITION p20200607 VALUES [('2020-06-07'), ('2020-06-08')),
PARTITION p20200608 VALUES [('2020-06-08'), ('2020-06-09')),
PARTITION p20200609 VALUES [('2020-06-09'), ('2020-06-10')),
PARTITION p20200610 VALUES [('2020-06-10'), ('2020-06-11')),
PARTITION p20200611 VALUES [('2020-06-11'), ('2020-06-12')),
PARTITION p20200612 VALUES [('2020-06-12'), ('2020-06-13')),
PARTITION p20200613 VALUES [('2020-06-13'), ('2020-06-14')),
PARTITION p20200614 VALUES [('2020-06-14'), ('2020-06-15')),
PARTITION p20200615 VALUES [('2020-06-15'), ('2020-06-16')),
PARTITION p20200616 VALUES [('2020-06-16'), ('2020-06-17')),
PARTITION p20200617 VALUES [('2020-06-17'), ('2020-06-18')),
PARTITION p20200618 VALUES [('2020-06-18'), ('2020-06-19')),
PARTITION p20200619 VALUES [('2020-06-19'), ('2020-06-20')),
PARTITION p20200620 VALUES [('2020-06-20'), ('2020-06-21')),
PARTITION p20200621 VALUES [('2020-06-21'), ('2020-06-22')),
PARTITION p20200622 VALUES [('2020-06-22'), ('2020-06-23')),
PARTITION p20200623 VALUES [('2020-06-23'), ('2020-06-24')),
PARTITION p20200624 VALUES [('2020-06-24'), ('2020-06-25')),
PARTITION p20200625 VALUES [('2020-06-25'), ('2020-06-26')),
PARTITION p20200626 VALUES [('2020-06-26'), ('2020-06-27')),
PARTITION p20200627 VALUES [('2020-06-27'), ('2020-06-28')),
PARTITION p20200628 VALUES [('2020-06-28'), ('2020-06-29')),
PARTITION p20200629 VALUES [('2020-06-29'), ('2020-06-30')),
PARTITION p20200630 VALUES [('2020-06-30'), ('2020-07-01')),
PARTITION p20200701 VALUES [('2020-07-01'), ('2020-07-02')),
PARTITION p20200702 VALUES [('2020-07-02'), ('2020-07-03')),
PARTITION p20200703 VALUES [('2020-07-03'), ('2020-07-04')),
PARTITION p20200705 VALUES [('2020-07-04'), ('2020-07-06')),
PARTITION p20200706 VALUES [('2020-07-06'), ('2020-07-07')),
PARTITION p20200707 VALUES [('2020-07-07'), ('2020-07-08')),
PARTITION p20200708 VALUES [('2020-07-08'), ('2020-07-09')),
PARTITION p20200709 VALUES [('2020-07-09'), ('2020-07-10')),
PARTITION p20200710 VALUES [('2020-07-10'), ('2020-07-11')),
PARTITION p20200711 VALUES [('2020-07-11'), ('2020-07-12')),
PARTITION p20200712 VALUES [('2020-07-12'), ('2020-07-13')),
PARTITION p20200713 VALUES [('2020-07-13'), ('2020-07-14')),
PARTITION p20200714 VALUES [('2020-07-14'), ('2020-07-15')),
PARTITION p20200715 VALUES [('2020-07-15'), ('2020-07-16')),
PARTITION p20200716 VALUES [('2020-07-16'), ('2020-07-17')),
PARTITION p20200717 VALUES [('2020-07-17'), ('2020-07-18')),
PARTITION p20200718 VALUES [('2020-07-18'), ('2020-07-19')),
PARTITION p20200719 VALUES [('2020-07-19'), ('2020-07-20')),
PARTITION p20200720 VALUES [('2020-07-20'), ('2020-07-21')),
PARTITION p20200721 VALUES [('2020-07-21'), ('2020-07-22')),
PARTITION p20200722 VALUES [('2020-07-22'), ('2020-07-23')),
PARTITION p20200723 VALUES [('2020-07-23'), ('2020-07-24')),
PARTITION p20200724 VALUES [('2020-07-24'), ('2020-07-25')),
PARTITION p20200725 VALUES [('2020-07-25'), ('2020-07-26')),
PARTITION p20200727 VALUES [('2020-07-26'), ('2020-07-28')),
PARTITION p20200728 VALUES [('2020-07-28'), ('2020-07-29')),
PARTITION p20200729 VALUES [('2020-07-29'), ('2020-07-30')),
PARTITION p20200730 VALUES [('2020-07-30'), ('2020-07-31')),
PARTITION p20200731 VALUES [('2020-07-31'), ('2020-08-01')),
PARTITION p20200801 VALUES [('2020-08-01'), ('2020-08-02')),
PARTITION p20200802 VALUES [('2020-08-02'), ('2020-08-03')),
PARTITION p20200803 VALUES [('2020-08-03'), ('2020-08-04')),
PARTITION p20200804 VALUES [('2020-08-04'), ('2020-08-05')),
PARTITION p20200805 VALUES [('2020-08-05'), ('2020-08-06')),
PARTITION p20200807 VALUES [('2020-08-06'), ('2020-08-08')),
PARTITION p20200808 VALUES [('2020-08-08'), ('2020-08-09')),
PARTITION p20200810 VALUES [('2020-08-09'), ('2020-08-11')),
PARTITION p20200811 VALUES [('2020-08-11'), ('2020-08-12')),
PARTITION p20200812 VALUES [('2020-08-12'), ('2020-08-13')),
PARTITION p20200813 VALUES [('2020-08-13'), ('2020-08-14')),
PARTITION p20200814 VALUES [('2020-08-14'), ('2020-08-15')),
PARTITION p20200815 VALUES [('2020-08-15'), ('2020-08-16')),
PARTITION p20200816 VALUES [('2020-08-16'), ('2020-08-17')),
PARTITION p20200817 VALUES [('2020-08-17'), ('2020-08-18')),
PARTITION p20200818 VALUES [('2020-08-18'), ('2020-08-19')),
PARTITION p20200819 VALUES [('2020-08-19'), ('2020-08-20')),
PARTITION p20200820 VALUES [('2020-08-20'), ('2020-08-21')),
PARTITION p20200821 VALUES [('2020-08-21'), ('2020-08-22')),
PARTITION p20200822 VALUES [('2020-08-22'), ('2020-08-23')),
PARTITION p20200823 VALUES [('2020-08-23'), ('2020-08-24')),
PARTITION p20200824 VALUES [('2020-08-24'), ('2020-08-25')),
PARTITION p20200825 VALUES [('2020-08-25'), ('2020-08-26')),
PARTITION p20200826 VALUES [('2020-08-26'), ('2020-08-27')),
PARTITION p20200827 VALUES [('2020-08-27'), ('2020-08-28')),
PARTITION p20200828 VALUES [('2020-08-28'), ('2020-08-29')),
PARTITION p20200829 VALUES [('2020-08-29'), ('2020-08-30')),
PARTITION p20200830 VALUES [('2020-08-30'), ('2020-08-31')),
PARTITION p20200831 VALUES [('2020-08-31'), ('2020-09-01')),
PARTITION p20200901 VALUES [('2020-09-01'), ('2020-09-02')),
PARTITION p20200902 VALUES [('2020-09-02'), ('2020-09-03')),
PARTITION p20200903 VALUES [('2020-09-03'), ('2020-09-04')),
PARTITION p20200904 VALUES [('2020-09-04'), ('2020-09-05')),
PARTITION p20200905 VALUES [('2020-09-05'), ('2020-09-06')),
PARTITION p20200906 VALUES [('2020-09-06'), ('2020-09-07')),
PARTITION p20200907 VALUES [('2020-09-07'), ('2020-09-08')),
PARTITION p20200908 VALUES [('2020-09-08'), ('2020-09-09')),
PARTITION p20200909 VALUES [('2020-09-09'), ('2020-09-10')),
PARTITION p20200910 VALUES [('2020-09-10'), ('2020-09-11')),
PARTITION p20200911 VALUES [('2020-09-11'), ('2020-09-12')),
PARTITION p20200912 VALUES [('2020-09-12'), ('2020-09-13')),
PARTITION p20200913 VALUES [('2020-09-13'), ('2020-09-14')),
PARTITION p20200914 VALUES [('2020-09-14'), ('2020-09-15')),
PARTITION p20200915 VALUES [('2020-09-15'), ('2020-09-16')),
PARTITION p20200916 VALUES [('2020-09-16'), ('2020-09-17')),
PARTITION p20200917 VALUES [('2020-09-17'), ('2020-09-18')),
PARTITION p20200918 VALUES [('2020-09-18'), ('2020-09-19')),
PARTITION p20200919 VALUES [('2020-09-19'), ('2020-09-20')),
PARTITION p20200920 VALUES [('2020-09-20'), ('2020-09-21')),
PARTITION p20200921 VALUES [('2020-09-21'), ('2020-09-22')),
PARTITION p20200922 VALUES [('2020-09-22'), ('2020-09-23')),
PARTITION p20200923 VALUES [('2020-09-23'), ('2020-09-24')),
PARTITION p20200924 VALUES [('2020-09-24'), ('2020-09-25')),
PARTITION p20200925 VALUES [('2020-09-25'), ('2020-09-26')),
PARTITION p20200926 VALUES [('2020-09-26'), ('2020-09-27')),
PARTITION p20200927 VALUES [('2020-09-27'), ('2020-09-28')),
PARTITION p20200928 VALUES [('2020-09-28'), ('2020-09-29')),
PARTITION p20200929 VALUES [('2020-09-29'), ('2020-09-30')),
PARTITION p20200930 VALUES [('2020-09-30'), ('2020-10-01')),
PARTITION p20201001 VALUES [('2020-10-01'), ('2020-10-02')),
PARTITION p20201002 VALUES [('2020-10-02'), ('2020-10-03')),
PARTITION p20201003 VALUES [('2020-10-03'), ('2020-10-04')),
PARTITION p20201004 VALUES [('2020-10-04'), ('2020-10-05')),
PARTITION p20201005 VALUES [('2020-10-05'), ('2020-10-06')),
PARTITION p20201006 VALUES [('2020-10-06'), ('2020-10-07')),
PARTITION p20201007 VALUES [('2020-10-07'), ('2020-10-08')),
PARTITION p20201008 VALUES [('2020-10-08'), ('2020-10-09')),
PARTITION p20201009 VALUES [('2020-10-09'), ('2020-10-10')),
PARTITION p20201010 VALUES [('2020-10-10'), ('2020-10-11')),
PARTITION p20201011 VALUES [('2020-10-11'), ('2020-10-12')),
PARTITION p20201012 VALUES [('2020-10-12'), ('2020-10-13')),
PARTITION p20201014 VALUES [('2020-10-13'), ('2020-10-15')),
PARTITION p20201015 VALUES [('2020-10-15'), ('2020-10-16')),
PARTITION p20201016 VALUES [('2020-10-16'), ('2020-10-17')),
PARTITION p20201017 VALUES [('2020-10-17'), ('2020-10-18')),
PARTITION p20201018 VALUES [('2020-10-18'), ('2020-10-19')),
PARTITION p20201019 VALUES [('2020-10-19'), ('2020-10-20')),
PARTITION p20201020 VALUES [('2020-10-20'), ('2020-10-21')),
PARTITION p20201021 VALUES [('2020-10-21'), ('2020-10-22')),
PARTITION p20201022 VALUES [('2020-10-22'), ('2020-10-23')),
PARTITION p20201023 VALUES [('2020-10-23'), ('2020-10-24')),
PARTITION p20201024 VALUES [('2020-10-24'), ('2020-10-25')),
PARTITION p20201025 VALUES [('2020-10-25'), ('2020-10-26')),
PARTITION p20201026 VALUES [('2020-10-26'), ('2020-10-27')),
PARTITION p20201027 VALUES [('2020-10-27'), ('2020-10-28')),
PARTITION p20201028 VALUES [('2020-10-28'), ('2020-10-29')),
PARTITION p20201029 VALUES [('2020-10-29'), ('2020-10-30')),
PARTITION p20201030 VALUES [('2020-10-30'), ('2020-10-31')),
PARTITION p20201031 VALUES [('2020-10-31'), ('2020-11-01')),
PARTITION p20201101 VALUES [('2020-11-01'), ('2020-11-02')),
PARTITION p20201102 VALUES [('2020-11-02'), ('2020-11-03')),
PARTITION p20201103 VALUES [('2020-11-03'), ('2020-11-04')),
PARTITION p20201104 VALUES [('2020-11-04'), ('2020-11-05')),
PARTITION p20201105 VALUES [('2020-11-05'), ('2020-11-06')),
PARTITION p20201106 VALUES [('2020-11-06'), ('2020-11-07')),
PARTITION p20201107 VALUES [('2020-11-07'), ('2020-11-08')),
PARTITION p20201108 VALUES [('2020-11-08'), ('2020-11-09')),
PARTITION p20201109 VALUES [('2020-11-09'), ('2020-11-10')),
PARTITION p20201111 VALUES [('2020-11-10'), ('2020-11-12')),
PARTITION p20201112 VALUES [('2020-11-12'), ('2020-11-13')),
PARTITION p20201113 VALUES [('2020-11-13'), ('2020-11-14')),
PARTITION p20201114 VALUES [('2020-11-14'), ('2020-11-15')),
PARTITION p20201115 VALUES [('2020-11-15'), ('2020-11-16')),
PARTITION p20201116 VALUES [('2020-11-16'), ('2020-11-17')),
PARTITION p20201117 VALUES [('2020-11-17'), ('2020-11-18')),
PARTITION p20201118 VALUES [('2020-11-18'), ('2020-11-19')),
PARTITION p20201119 VALUES [('2020-11-19'), ('2020-11-20')),
PARTITION p20201120 VALUES [('2020-11-20'), ('2020-11-21')),
PARTITION p20201121 VALUES [('2020-11-21'), ('2020-11-22')),
PARTITION p20201122 VALUES [('2020-11-22'), ('2020-11-23')),
PARTITION p20201123 VALUES [('2020-11-23'), ('2020-11-24')),
PARTITION p20201124 VALUES [('2020-11-24'), ('2020-11-25')),
PARTITION p20201125 VALUES [('2020-11-25'), ('2020-11-26')),
PARTITION p20201126 VALUES [('2020-11-26'), ('2020-11-27')),
PARTITION p20201127 VALUES [('2020-11-27'), ('2020-11-28')),
PARTITION p20201128 VALUES [('2020-11-28'), ('2020-11-29')),
PARTITION p20201129 VALUES [('2020-11-29'), ('2020-11-30')))
DISTRIBUTED BY HASH(`bow_id`) BUCKETS 32
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "DEFAULT"
);
```
2. Starting broker load job as many as you can.
3. Using `jstat` and `jmap` to observe the jvm.
4. Using `show proc '/transactions/db_id';` to ovserve Doris load job count.
**Expected behavior**
The old space of heap continues to grow, eventually leading to continued full gc.
At this time, Doris FE will be unavailable.
**Screenshots**
`jstat -gccause fe_pid` output screenshot:
<img width="843" alt="gc_cause" src="https://user-images.githubusercontent.com/26450235/100712415-ad23c380-33ed-11eb-86ad-f26b6819e656.png">
Here is an example of unavailable FE's heap snapshot:
<img width="703" alt="heap_snapshot" src="https://user-images.githubusercontent.com/26450235/100709894-d5a9be80-33e9-11eb-8243-9a377ebaa658.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 closed issue #4992: [bug] finished load jobs cost too much heap memory
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 closed issue #4992:
URL: https://github.com/apache/incubator-doris/issues/4992
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org