You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/09/26 15:00:09 UTC
[jira] [Commented] (FLINK-7668) Add AccessExecutionGraph refresh
interval to ExecutionGraphHolder
[ https://issues.apache.org/jira/browse/FLINK-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180895#comment-16180895 ]
ASF GitHub Bot commented on FLINK-7668:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/4728
[FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers
## What is the purpose of the change
The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the latter, the former
does not expect the AccessExecutionGraph to be the true ExecutionGraph. Instead it assumes
it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache entries after
a given time to live period. This will trigger requesting the AccessExecutionGraph again
and, thus, updating the ExecutionGraph information for the ExecutionGraph based REST
handlers.
In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic cleanup task
which triggers ExecutionGraphCache.cleanup. This methods releases all cache entries which
have exceeded their time to live. Currently it is set to 20 * refreshInterval of the
web gui.
## Brief change log
- Rename `ExecutionGraphHolder` into `ExecutionGraphCache`
- Make `ExecutionGraphCache` thread safe by using a `ConcurrentHashMap`
- Add `ExecutionGraph` request logic depending on the time to live for each cache entry
- Add periodic cleanup task to `WebRuntimeMonitor` which triggers `ExecutionGraphCache#cleanup`
## Verifying this change
This change added tests and can be verified as follows:
- `ExecutionGraphCacheTest` adds multiple tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink refreshExecutionGraphHolder
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4728.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4728
----
commit ed2d5482c315d64216e4a645b68e7b48dfb89456
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-09-21T08:53:24Z
[FLINK-7650] [flip6] Port JobCancellationHandler to new REST endpoint
Let the JobCancellationHandler implement the LegacyRestHandler interface. Moreover,
this commit adds the DELETE method to HttpMethodWrapper and the
RestServerEndpoint#registerHandler method.
commit e83065fa8102c10a10cdc981f0c7608577c20c34
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-09-21T14:47:18Z
[FLINK-7649] [flip6] Extend JobTerminationHandler to support stop
Rename the JobCancellationHandler into JobTerminationHandler which is now responsible
for terminating jobs. Moreover, this commits adds two termination modes, cancel and stop,
which are specified by providing a query parameter.
commit 02e345b8f34890d67230d0a75a818a8e3b65cf2b
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-09-22T11:31:12Z
[FLINK-7667] [flip6] Use ArchivedExecutionGraph as serializable AccessExecutionGraph
This commit removes AccessExecutionGraph#getCheckpointCoordinator and changes the
AccessExecutionGraph#getJobCheckpointSettings into #getJobCheckpointConfiguration.
The JobCheckpointConfiguration only contains the CheckpointCoordinator relevant
configuration settings and excludes the serialized state backend and the
serialized master hooks. That way we don't send unnecessary information when
the ArchivedExecutionGraph is requested.
commit ab4b21fdf0c00d531d04de6c6272c2812cb61e51
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-09-25T13:29:59Z
[FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers
The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the latter, the former
does not expect the AccessExecutionGraph to be the true ExecutionGraph. Instead it assumes
it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache entries after
a given time to live period. This will trigger requesting the AccessExecutionGraph again
and, thus, updating the ExecutionGraph information for the ExecutionGraph based REST
handlers.
In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic cleanup task
which triggers ExecutionGraphCache.cleanup. This methods releases all cache entries which
have exceeded their time to live. Currently it is set to 20 * refreshInterval of the
web gui.
----
> Add AccessExecutionGraph refresh interval to ExecutionGraphHolder
> -----------------------------------------------------------------
>
> Key: FLINK-7668
> URL: https://issues.apache.org/jira/browse/FLINK-7668
> Project: Flink
> Issue Type: Sub-task
> Components: REST
> Affects Versions: 1.4.0
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Labels: flip-6
>
> Once we support offline {{AccessExecutionGraph}} implementation (see FLINK-7667) we should add a refresh interval to the {{ExecutionGraphHolder}} after which the {{AccessExecutionGraph}} is retrieved again from the {{JobMaster}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)