You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "hujiahua (Jira)" <ji...@apache.org> on 2021/11/17 02:27:00 UTC
[jira] [Created] (KYLIN-5128) The job of resizing global dict bucket sometimes run for a long time
hujiahua created KYLIN-5128:
-------------------------------
Summary: The job of resizing global dict bucket sometimes run for a long time
Key: KYLIN-5128
URL: https://issues.apache.org/jira/browse/KYLIN-5128
Project: Kylin
Issue Type: Improvement
Affects Versions: v4.0.0
Reporter: hujiahua
Attachments: image-2021-11-17-10-03-26-943.png, image-2021-11-17-10-12-46-187.png
I often encounter cube building job running for a long time in the global dict resizing process stage. After spark stage analysis, I found that it was caused by too little concurrency of the task.
!image-2021-11-17-10-03-26-943.png!
And I also found kylin using sparkSession.createDataset to build dict bucket dataset, where mean the parallelize size was `sparkContext.defaultParallelism`. When enable spark executor dynamic allocation (spark.dynamicAllocation.enabled) ,sparkContext.defaultParallelism will change during runtime, and have a chance to get a small parallelism value.
!image-2021-11-17-10-12-46-187.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re:[jira] [Created] (KYLIN-5128) The job of resizing global dict bucket sometimes run for a long time
Posted by 大数据开发工程师-付德彬 <bi...@163.com>.
退订
| |
大数据开发工程师-付德彬
|
|
bingclouds@163.com
|
签名由网易邮箱大师定制
On 11/17/2021 10:27，hujiahua (Jira)<ji...@apache.org> wrote：
hujiahua created KYLIN-5128:
-------------------------------
Summary: The job of resizing global dict bucket sometimes run for a long time
Key: KYLIN-5128
URL: https://issues.apache.org/jira/browse/KYLIN-5128
Project: Kylin
Issue Type: Improvement
Affects Versions: v4.0.0
Reporter: hujiahua
Attachments: image-2021-11-17-10-03-26-943.png, image-2021-11-17-10-12-46-187.png
I often encounter cube building job running for a long time in the global dict resizing process stage. After spark stage analysis, I found that it was caused by too little concurrency of the task.
!image-2021-11-17-10-03-26-943.png!
And I also found kylin using sparkSession.createDataset to build dict bucket dataset, where mean the parallelize size was `sparkContext.defaultParallelism`. When enable spark executor dynamic allocation (spark.dynamicAllocation.enabled) ,sparkContext.defaultParallelism will change during runtime, and have a chance to get a small parallelism value.
!image-2021-11-17-10-12-46-187.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)