You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Yida Wu (Code Review)" <ge...@cloudera.org> on 2025/02/27 03:48:23 UTC
[Impala-ASF-CR] IMPALA-13798: Cleanup host-level remote scratch dir on shutdown
Yida Wu has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/22549 )
Change subject: IMPALA-13798: Cleanup host-level remote scratch dir on shutdown
......................................................................
IMPALA-13798: Cleanup host-level remote scratch dir on shutdown
IMPALA-13677 introduces cleanup of remote scratch files on startup,
and also introduces a host-level directory to do easier cleanup.
An empty host-level directory in remote storage does not use
resources, but it can stay there forever if a host goes offline
permanently or hostname changes. To improve this behavior, this
patch adds support for removing the remote host-level temporary
directory on shutdown. This helps prevent too many empty
directories left in the remote scratch path, especially for hosts
that never restart.
Changed the flag remote_scratch_cleanup_on_startup to
remote_scratch_cleanup_on_start_stop, so that this flag also
controls cleanup on shutdown.
Tests:
Passed exhaustive tests.
Added an e2e testcase
test_scratch_dirs_remote_dir_removal_on_shutdown.
Change-Id: Ic8f446894afdf975630aef80a9d964a9a78d3b46
---
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M be/src/service/impala-server.cc
M tests/custom_cluster/test_scratch_disk.py
6 files changed, 87 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22549/3
--
To view, visit http://gerrit.cloudera.org:8080/22549
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic8f446894afdf975630aef80a9d964a9a78d3b46
Gerrit-Change-Number: 22549
Gerrit-PatchSet: 3
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>