[jira] [Commented] (KYLIN-5363) Segment build task cannot be terminated for too large parallel tasks size

["pengfei.zhan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647076#comment-17647076 ] 

pengfei.zhan commented on KYLIN-5363:
-------------------------------------

The metadata of the segment build task is as follows:
 !5363-1.png! 

The folded field `tasks` has three sub-tasks which means:
* resource detection, detecting cores and memory can be used by this job;
* loading data, building segment;
* updating metadata, updating some metadata like table, snapshot, etc.

The second step was split into 8 stages. Each of them has its own status. These statuses were recorded in the field of `stagesMap`.
 !5363-2.png! 

What happens when terminating segment-built tasks? The program will traverse the `stageMap` to get all stages, then update the status of each stage. However, the metadata was copied and updated. If the segment tasks are very large, the usage of memory will creep up in a short time. 

We found that this doesn't only happen in the method of `updateStageStatus`, the method of `updateJobOutput` has this problem too. 






> Segment build task cannot be terminated for too large parallel tasks size
> -------------------------------------------------------------------------
>
>                 Key: KYLIN-5363
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5363
>             Project: Kylin
>          Issue Type: Bug
>          Components: Job Engine
>    Affects Versions: 5.0-alpha
>            Reporter: pengfei.zhan
>            Assignee: pengfei.zhan
>            Priority: Major
>             Fix For: 5.0-alpha
>
>         Attachments: 5363-1.png, 5363-2.png
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)