[jira] [Commented] (IMPALA-13677) Cleanup of s3 scratch files on abnormal executor exit

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931815#comment-17931815 ] 

ASF subversion and git services commented on IMPALA-13677:
----------------------------------------------------------

Commit ddd4f4f8d68addce1542d57f94c637210a090150 in impala's branch refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ddd4f4f8d ]

IMPALA-13798: Cleanup host-level remote scratch dir on shutdown

IMPALA-13677 introduces cleanup of remote scratch files on startup,
and also introduces a host-level directory to do easier cleanup.

An empty host-level directory in remote storage does not use
resources, but it can stay there forever if a host goes offline
permanently or hostname changes. To improve this behavior, this
patch adds support for removing the remote host-level temporary
directory on shutdown. This helps prevent too many empty
directories left in the remote scratch path, especially for hosts
that never restart.

Changed the flag remote_scratch_cleanup_on_startup to
remote_scratch_cleanup_on_start_stop, so that this flag also
controls cleanup on shutdown.

Tests:
Passed exhaustive tests.
Added an e2e testcase
test_scratch_dirs_remote_dir_removal_on_shutdown.

Change-Id: Ic8f446894afdf975630aef80a9d964a9a78d3b46
Reviewed-on: http://gerrit.cloudera.org:8080/22549
Reviewed-by: Daniel Becker <da...@cloudera.com>
Reviewed-by: Abhishek Rawat <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Cleanup of s3 scratch files on abnormal executor exit
> -----------------------------------------------------
>
>                 Key: IMPALA-13677
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13677
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>            Reporter: Yida Wu
>            Assignee: Yida Wu
>            Priority: Critical
>             Fix For: Impala 4.5.0
>
>
> Currently, when an executor spills data to a remote storage, scratch files would remain in the remote storage if the executor exits abnormally or is terminated after the graceful shutdown deadline.
> Immediate removal may be challenging, and no concrete solution is currently available. However, to clear leftover files on startup is possible.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org