You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "sercanCyberVision (via GitHub)" <gi...@apache.org> on 2023/06/05 14:08:57 UTC
[GitHub] [hive] sercanCyberVision commented on a diff in pull request #4293: HIVE-27317: Temporary (local) session files cleanup improvements
sercanCyberVision commented on code in PR #4293:
URL: https://github.com/apache/hive/pull/4293#discussion_r1218129590
##########
ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java:
##########
@@ -236,4 +243,29 @@ static Options createOptions() {
return result;
}
+
+ /**
+ * While deleting dangling scratch dirs from hdfs, we can clean corresponding local files as well
+ * @param sessionName prefix to determine removable tmp files
+ * @param localTmpdir local tmp file location
+ */
+ public void removeLocalTmpFiles(String sessionName, String localTmpdir) {
+ File[] files = new File(localTmpdir).listFiles();
+ boolean success;
+ if (files != null) {
+ for (File file : files) {
+ success = false;
+ if (file.getName().startsWith(sessionName)) {
Review Comment:
I have moved the constructor to the first place where `localTmpdir` is created (out of the loops). It will be invoked once per service run.
##########
ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java:
##########
@@ -236,4 +243,29 @@ static Options createOptions() {
return result;
}
+
+ /**
+ * While deleting dangling scratch dirs from hdfs, we can clean corresponding local files as well
+ * @param sessionName prefix to determine removable tmp files
+ * @param localTmpdir local tmp file location
+ */
+ public void removeLocalTmpFiles(String sessionName, String localTmpdir) {
+ File[] files = new File(localTmpdir).listFiles();
+ boolean success;
+ if (files != null) {
+ for (File file : files) {
+ success = false;
+ if (file.getName().startsWith(sessionName)) {
+ success = file.delete();
Review Comment:
I have added `File.canWrite()` check before attempting to delete.
##########
ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java:
##########
@@ -236,4 +243,29 @@ static Options createOptions() {
return result;
}
+
+ /**
+ * While deleting dangling scratch dirs from hdfs, we can clean corresponding local files as well
+ * @param sessionName prefix to determine removable tmp files
+ * @param localTmpdir local tmp file location
+ */
+ public void removeLocalTmpFiles(String sessionName, String localTmpdir) {
Review Comment:
@veghlaci05 this is visible for testing. I have added `@VisibleForTesting` annotation. Initially the method was private and in unit test I have executed whole service, but service failed with below exception;
```
java.lang.UnsupportedOperationException: Append is not supported by ChecksumFileSystem
```
at below [point](https://github.com/apache/hive/blob/badba5e8a3b98f8c7be4903a6d9b62f8ea5e142e/ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java#L142);
```
IOUtils.closeStream(fs.append(lockFilePath));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org