[jira] [Assigned] (KYLIN-5636) automatically clean up dependent files after the build task

["Zhiting Guo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhiting Guo reassigned KYLIN-5636:
----------------------------------

    Assignee: Zhiting Guo

> automatically clean up dependent files after the build task
> -----------------------------------------------------------
>
>                 Key: KYLIN-5636
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5636
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Tools, Build and Test
>    Affects Versions: 5.0-alpha
>            Reporter: Zhiting Guo
>            Assignee: Zhiting Guo
>            Priority: Major
>             Fix For: 5.0-alpha
>
>
> *question:*
> The files uploaded under the path spark.kubernetes.file.upload.path are not automatically deleted
> 1: When spark creates a driverPod, it uploads dependencies to the specified path. The build task is in cluster mode and needs to create a driverPod. Running the build task multiple times results in a large path file.
> 2: At present, the upload.path path we configured (s3a://kylin/spark-on-k8s) is a fixed path, and spark will create a subdirectory in this directory, the spark-upload-uuid directory, and then store the dependencies in it.
> *dev design*
> Core idea, add dynamic subdirectory under the original upload.path path, delete the entire subdirectory when the task is over
> Build task: upload.path + jobId (e.g. s3a://kylin/spark-on-k8s/uuid)
> Delete the dependency directory when the build task is finished
>  
> Automatically delete dependent function is called, kill-9 situation will lead to the deletion function is not called, garbage cleaning function needs to be added to the bottom of the policy, such as greater than three months before the directory is automatically deleted



--
This message was sent by Atlassian Jira
(v8.20.10#820010)