[GitHub] [flink] zentol commented on a change in pull request #12636: [FLINK-17998][hs] Support Flink history server archive size limitation

[GitBox <gi...@apache.org>]

zentol commented on a change in pull request #12636:
URL: https://github.com/apache/flink/pull/12636#discussion_r440115913



##########
File path: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
##########
@@ -295,6 +315,19 @@ public void run() {
 			}
 		}
 
+		private List<ArchiveEvent> cleanupJobsBeyondSizeLimit(Set<Path> jobArchivesToRemove) {
+			Set<String> jobIdsToRemoveFromOverview = new HashSet<>();
+			for (Path archive : jobArchivesToRemove) {
+				jobIdsToRemoveFromOverview.add(archive.getName());
+				try {
+					archive.getFileSystem().delete(archive, false);
+				} catch (IOException ioe) {
+					LOG.error("Error while deleting old archive " + archive, ioe);
+				}
+			}
+			return cleanupExpiredJobs(jobIdsToRemoveFromOverview);

Review comment:
       This is redundant; cleanupExpiredJobs already handles this case since we're skipping `jobsToRemove.remove(jobID);` at L235.

##########
File path: flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java
##########
@@ -99,6 +99,11 @@
 			.withDescription("Enable HTTPs access to the HistoryServer web frontend. This is applicable only when the" +
 				" global SSL flag security.ssl.enabled is set to true.");
 
+	public static final ConfigOption<Integer> HISTORY_SERVER_MAX_SIZE =
+		key("historyserver.max.history.size")
+			.defaultValue(50)

Review comment:
       We can't enable this by default since this could result in us deleting externally kept data all of a sudden.

##########
File path: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
##########
@@ -205,6 +212,10 @@ public void run() {
 					if (jobArchives == null) {
 						continue;
 					}
+
+					Arrays.sort(jobArchives, Comparator.comparingLong(FileStatus::getModificationTime).reversed());

Review comment:
       It should be documented that the `maxHistorySize` is enforced per archive directory, and not globally.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org