You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zentol <gi...@git.apache.org> on 2017/02/08 10:45:59 UTC
[GitHub] flink pull request #3286: [FLINK-1579] [WIP] Implement Standalone HistorySer...
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/3286
[FLINK-1579] [WIP] Implement Standalone HistoryServer
This PR is a work-in-progress view over a standalone History Server (HS).
JobManagers may send completed jobs to the HistoryServer for them to be archived. Upon receiving an ArchivedExecutionGraph the HS pre-computes all possible REST requests and writes them into files. The files are arranged in a directory structure corresponding to the REST API.
The HS can be started by calling `./bin/historyserver.sh start`, similar to the JM/TM. Various config options exist for the HS that mostly mirror the web-ui/RPC options of the JM.
The HS uses a slightly modified web-ui; basically it only shows the "Completed Jobs" page. To not duplicate the everything I've added 2 files, `index2.jade` and `index2.coffee`, to the build script. The resulting `index2.html` file will be loaded when the browser requests the `index.html`.
In order to re-use the JSON generation code that previously was contained in various handlers a giant utility `JsonUtils` class was created. This class now contains a variety of static methods that generate the JSON responses. As a result most handlers were reduced to one-liners, bar some sanity-checks.
In regard to tests we verify that the HS creates all expected files upon receiving an ExecutionGraph.
Furthermore, the newly created JsonUtils are mostly tested (the new checkpoint stats aren't tested); so we have tests for the REST responses now, which is neat.
I'm not opening a proper PR yet as i have to go through all changes once again in detail, but it works (locally and on a cluster) so i wanted people to try it out and get some feedback.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 1579_history_server_b
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3286.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3286
----
commit 0fdfeec0c86cba60d271d38cfbce7e4ae759b700
Author: zentol <ch...@apache.org>
Date: 2016-10-17T10:55:19Z
Add AccessExecutionVertex#getPriorExecutions()
commit 18c4cc6a9e8f3c9b772bcfe8f866e07d2f7304ce
Author: zentol <ch...@apache.org>
Date: 2017-01-30T15:06:13Z
[FLINK-5645] EG stuff
commit fcc4def5251086d4e37901c58bc47785e1d90788
Author: zentol <ch...@apache.org>
Date: 2017-01-24T09:13:24Z
[FLINK-1579] Implement History Server - Frontend
commit 2cc6b736c0c5c78903b85f9c1a9ccde8c3ee70b8
Author: zentol <ch...@apache.org>
Date: 2016-10-21T12:29:30Z
[FLINK-1579] Implement History Server - Backend
commit 0047ae53b9f2f79eee9ec7e76195559b32dbeb20
Author: zentol <ch...@apache.org>
Date: 2017-02-08T08:58:01Z
[FLINK-1579] Implement History Server - Backend - Tests
commit 730548a7d88c56a2cde235e3d7d92dbf676611b7
Author: zentol <ch...@apache.org>
Date: 2017-02-08T08:58:22Z
Use JsonUtils in handlers
commit adcc161e46f817e80301d1fb885cdef4a8679a71
Author: zentol <ch...@apache.org>
Date: 2017-02-08T10:23:56Z
Rebuild web-frontend
commit 3227fc2a12e8aeaaf111339833123da708ccea70
Author: zentol <ch...@apache.org>
Date: 2017-02-08T10:24:14Z
tmp streaming example with checkpointing
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3286: [FLINK-1579] [WIP] Implement Standalone HistoryServer
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the issue:
https://github.com/apache/flink/pull/3286
Thanks for this PR. I think it's super cool (and long overdue) to finally have this. I think the general approach is sound and has good things going for it.
But I see two problems with maintaining the current implementation in the long run:
- Changing a URL in the web dashboard requires that we also change it in the history server file generation. It's easy to accidentally break the history server and it's cumbersome to write tests when we extend it.
- We have all JSON generation in the JsonUtil. That results in a giant class (as you say) that is really tightly coupled to the handlers but resides in a different place.
I think we can word around this by moving the archival logic to the handlers. I think it's fair that handlers that want to take part in historical archiving implement their archival logic on their own. The handlers would need to return their path and also their JSON. That way we keep all the handler logic in one place for both the regular frontend and history server. A "historic execution graph" might be a simple collection of `path, JSON` entries that the history server persists on receive.
Another minor thing is that the final status of the operators is missing on the history server. I don't know whether that is a front end issue or something is missing in the JSON.
I think it's best to close this PR for now and think about how we want to extend this a bit for better maintainability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3286: [FLINK-1579] [WIP] Implement Standalone HistorySer...
Posted by zentol <gi...@git.apache.org>.
Github user zentol closed the pull request at:
https://github.com/apache/flink/pull/3286
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---