You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "eemario (via GitHub)" <gi...@apache.org> on 2026/01/16 09:05:15 UTC
[PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
eemario opened a new pull request, #27430:
URL: https://github.com/apache/flink/pull/27430
## What is the purpose of the change
This pull request make HistoryServer support application archives while ensuring that legacy job archives are handled properly.
## Brief change log
- Add configuration options for application archive fetching
- Adapt the archive retention strategy to be compatible with application archives
- Introduce a new archiveFetcher for application archives (and their associated job archives)
- Integrate the new archiveFetcher into the HistoryServer
## Verifying this change
This change added tests and can be verified as follows:
- Updated tests for archive retention strategy to include coverage for application archives
- Added tests that validates the behavior of the HistoryServer when fetching jobs and applications
- Manually verified the change by running a standalone cluster and HistoryServer, confirming that the archives can be parsed properly and served via REST responses
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2697668845
##########
docs/layouts/shortcodes/generated/history_server_configuration.html:
##########
@@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
+ <tr>
+ <td><h5>historyserver.archive.retained-applications</h5></td>
+ <td style="word-wrap: break-word;">-1</td>
+ <td>Integer</td>
+ <td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consi
stent across them. </li></ul></td>
Review Comment:
What is org.apache.flink.configuration.description.TextElement@ae3540e - is this supposed to be a link?
what is <br />. This seems to be an end br tag with a space in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2703068406
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
+ private static final String LEGACY_NOTE_MESSAGE =
+ "Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
+ private static final String RETAINED_STRATEGY_MESSAGE =
+ "Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";
+
+ /** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
+ public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
+ key("historyserver.archive.clean-expired-jobs")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
+ code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
+ .linebreak()
+ .text(LEGACY_NOTE_MESSAGE)
Review Comment:
Thank you for raising this point. After some thought, I believe there are some nuances worth clarifying regarding the History Server.
These configuration options specifically apply to job archives generated prior to FLINK-38761 (most likely before Flink 2.4). Until that legacy format reaches its end-of-service, the History Server must remain capable of parsing such archives and providing users with the necessary configuration parameters to control their behavior. Therefore, these options are not deprecated and will not be removed in the next major version.
That said, if you feel this approach is problematic or have concerns about maintaining these options, I’m happy to discuss further!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2697709946
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
+ private static final String LEGACY_NOTE_MESSAGE =
+ "Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
+ private static final String RETAINED_STRATEGY_MESSAGE =
+ "Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";
+
+ /** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
+ public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
+ key("historyserver.archive.clean-expired-jobs")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
+ code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
+ .linebreak()
+ .text(LEGACY_NOTE_MESSAGE)
Review Comment:
we should deprecate the options that only apply to the legacy case. I assume there is an intention to remove them in the next version change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2710726417
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
+ private static final String LEGACY_NOTE_MESSAGE =
+ "Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
+ private static final String RETAINED_STRATEGY_MESSAGE =
+ "Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";
+
+ /** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
+ public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
+ key("historyserver.archive.clean-expired-jobs")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
+ code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
+ .linebreak()
+ .text(LEGACY_NOTE_MESSAGE)
Review Comment:
Hi @davidradl I'd appreciate a quick confirmation—are there any remaining concerns or questions from your side? If not, the PR should be ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2707188878
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerApplicationArchiveFetcher.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.webmonitor.history;
+
+import org.apache.flink.api.common.ApplicationID;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.HistoryServerOptions;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.history.ArchivePathUtils;
+import org.apache.flink.runtime.history.FsJsonArchivist;
+import org.apache.flink.runtime.messages.webmonitor.ApplicationDetails;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleApplicationsDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.rest.messages.ApplicationsOverviewHeaders;
+import org.apache.flink.runtime.rest.messages.JobsOverviewHeaders;
+import org.apache.flink.runtime.webmonitor.history.retaining.ArchiveRetainedStrategy;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.jackson.JacksonMapperFactory;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonFactory;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is used by the {@link HistoryServer} to fetch the application and job archives that
+ * are located at {@link HistoryServerOptions#HISTORY_SERVER_ARCHIVE_DIRS}. The directories are
+ * polled in regular intervals, defined by {@link
+ * HistoryServerOptions#HISTORY_SERVER_ARCHIVE_REFRESH_INTERVAL}.
+ *
+ * <p>The archives are downloaded and expanded into a file structure analog to the REST API.
+ *
+ * <p>Removes existing archives from these directories and the cache according to {@link
+ * ArchiveRetainedStrategy} and {@link
+ * HistoryServerOptions#HISTORY_SERVER_CLEANUP_EXPIRED_APPLICATIONS}.
+ */
+public class HistoryServerApplicationArchiveFetcher {
Review Comment:
Updated: Make `HistoryServerApplicationArchiveFetcher` extends `HistoryServerArchiveFetcher` to reuse the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2697668845
##########
docs/layouts/shortcodes/generated/history_server_configuration.html:
##########
@@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
+ <tr>
+ <td><h5>historyserver.archive.retained-applications</h5></td>
+ <td style="word-wrap: break-word;">-1</td>
+ <td>Integer</td>
+ <td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consi
stent across them. </li></ul></td>
Review Comment:
What is org.apache.flink.configuration.description.TextElement@ae3540e - is this supposed to be a link?
what is `<br />`. This seems to be an end br tag with a space in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2703068406
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
+ private static final String LEGACY_NOTE_MESSAGE =
+ "Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
+ private static final String RETAINED_STRATEGY_MESSAGE =
+ "Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";
+
+ /** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
+ public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
+ key("historyserver.archive.clean-expired-jobs")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
+ code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
+ .linebreak()
+ .text(LEGACY_NOTE_MESSAGE)
Review Comment:
Hi @davidradl Thank you for raising this point. After some thought, I believe there are some nuances worth clarifying regarding the History Server.
These configuration options specifically apply to job archives generated prior to FLINK-38761 (most likely before Flink 2.3). Until that legacy format reaches its end-of-service, the History Server must remain capable of parsing such archives and providing users with the necessary configuration parameters to control their behavior. Therefore, these options are not deprecated and will not be removed in the next major version.
That said, if you feel this approach is problematic or have concerns about maintaining these options, I’m happy to discuss further!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2697668845
##########
docs/layouts/shortcodes/generated/history_server_configuration.html:
##########
@@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
+ <tr>
+ <td><h5>historyserver.archive.retained-applications</h5></td>
+ <td style="word-wrap: break-word;">-1</td>
+ <td>Integer</td>
+ <td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consi
stent across them. </li></ul></td>
Review Comment:
What is org.apache.flink.configuration.description.TextElement@ae3540e - is this supposed to be a link?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2697704881
##########
docs/layouts/shortcodes/generated/history_server_configuration.html:
##########
@@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
+ <tr>
+ <td><h5>historyserver.archive.retained-applications</h5></td>
+ <td style="word-wrap: break-word;">-1</td>
+ <td>Integer</td>
+ <td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consi
stent across them. </li></ul></td>
Review Comment:
This was caused by a mistake in the HistoryServerOptions description. I will fix it and regenerate the docs. Thanks for pointing it out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2703068406
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
+ private static final String LEGACY_NOTE_MESSAGE =
+ "Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
+ private static final String RETAINED_STRATEGY_MESSAGE =
+ "Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";
+
+ /** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
+ public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
+ key("historyserver.archive.clean-expired-jobs")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
+ code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
+ .linebreak()
+ .text(LEGACY_NOTE_MESSAGE)
Review Comment:
Hi @davidradl Thank you for raising this point. After some thought, I believe there are some nuances worth clarifying regarding the History Server.
These configuration options specifically apply to job archives generated prior to FLINK-38761 (most likely before Flink 2.4). Until that legacy format reaches its end-of-service, the History Server must remain capable of parsing such archives and providing users with the necessary configuration parameters to control their behavior. Therefore, these options are not deprecated and will not be removed in the next major version.
That said, if you feel this approach is problematic or have concerns about maintaining these options, I’m happy to discuss further!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2706596330
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerApplicationArchiveFetcher.java:
##########
@@ -0,0 +1,577 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.webmonitor.history;
+
+import org.apache.flink.api.common.ApplicationID;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.HistoryServerOptions;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.history.ArchivePathUtils;
+import org.apache.flink.runtime.history.FsJsonArchivist;
+import org.apache.flink.runtime.messages.webmonitor.ApplicationDetails;
+import org.apache.flink.runtime.messages.webmonitor.JobDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleApplicationsDetails;
+import org.apache.flink.runtime.messages.webmonitor.MultipleJobsDetails;
+import org.apache.flink.runtime.rest.messages.ApplicationsOverviewHeaders;
+import org.apache.flink.runtime.rest.messages.JobsOverviewHeaders;
+import org.apache.flink.runtime.webmonitor.history.retaining.ArchiveRetainedStrategy;
+import org.apache.flink.util.FileUtils;
+import org.apache.flink.util.jackson.JacksonMapperFactory;
+
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonFactory;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Consumer;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This class is used by the {@link HistoryServer} to fetch the application and job archives that
+ * are located at {@link HistoryServerOptions#HISTORY_SERVER_ARCHIVE_DIRS}. The directories are
+ * polled in regular intervals, defined by {@link
+ * HistoryServerOptions#HISTORY_SERVER_ARCHIVE_REFRESH_INTERVAL}.
+ *
+ * <p>The archives are downloaded and expanded into a file structure analog to the REST API.
+ *
+ * <p>Removes existing archives from these directories and the cache according to {@link
+ * ArchiveRetainedStrategy} and {@link
+ * HistoryServerOptions#HISTORY_SERVER_CLEANUP_EXPIRED_APPLICATIONS}.
+ */
+public class HistoryServerApplicationArchiveFetcher {
Review Comment:
Is it possible to reuse the logic of `HistoryServerArchiveFetcher`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk merged PR #27430:
URL: https://github.com/apache/flink/pull/27430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27430:
URL: https://github.com/apache/flink/pull/27430#issuecomment-3758898507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "98c47df8297543da021af9c57945b62b28fa10d7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c47df8297543da021af9c57945b62b28fa10d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c47df8297543da021af9c57945b62b28fa10d7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2697668845
##########
docs/layouts/shortcodes/generated/history_server_configuration.html:
##########
@@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
+ <tr>
+ <td><h5>historyserver.archive.retained-applications</h5></td>
+ <td style="word-wrap: break-word;">-1</td>
+ <td>Integer</td>
+ <td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consi
stent across them. </li></ul></td>
Review Comment:
What is org.apache.flink.configuration.description.TextElement@ae3540e - is this supposed to be a link?
what is `<br />`. This seems to be an end br tag with a space in. what does this mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38777][history] HistoryServer supports application archives [flink]
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk commented on code in PR #27430:
URL: https://github.com/apache/flink/pull/27430#discussion_r2704456379
##########
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/history/HistoryServerTest.java:
##########
@@ -130,6 +149,9 @@ void testHistoryServerIntegration(final boolean versionLessThan14) throws Except
== HistoryServerArchiveFetcher.ArchiveEventType.CREATED) {
numExpectedArchivedJobs.countDown();
}
+ },
+ (event) -> {
+ throw new RuntimeException("Should not called");
Review Comment:
called -> call
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -164,6 +171,52 @@ public class HistoryServerOptions {
code("IllegalConfigurationException"))
.linebreak()
.text(NOTE_MESSAGE)
+ .list(
+ text(CONFIGURE_SINGLE_INSTANCE),
+ text(CONFIGURE_CONSISTENT))
+ .linebreak()
+ .text(LEGACY_NOTE_MESSAGE)
+ .build());
+
+ /**
+ * If this option is enabled then deleted application archives are also deleted from
+ * HistoryServer.
+ */
+ public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_APPLICATIONS =
+ key("historyserver.archive.clean-expired-applications")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
Review Comment:
jobs -> applications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org