You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by kbialek <gi...@git.apache.org> on 2018/02/19 21:30:52 UTC
[GitHub] flink pull request #5530: [FLINK-8660] Enable the user to provide custom HAS...
GitHub user kbialek opened a pull request:
https://github.com/apache/flink/pull/5530
[FLINK-8660] Enable the user to provide custom HAServices implementation
## What is the purpose of the change
This pull request makes it possible to use alternative (to ZK) HA backend implementation
## Brief change log
* Added HighAvailabilityServicesFactory interface
* Added new option HighAvailabilityMode.CUSTOM
* Added new configuration property high-availability.factoryClass
* HighAvailabilityServicesUtils uses configured factory to instantiate HighAvailabilityServices
* Updated docs
## Verifying this change
This change added tests and can be verified as follows:
* Added new test class HighAvailabilityServicesUtilsTest
* Updated test class HighAvailabilityModeTest
* Manually verified by using own HA backend implementation
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes, if this feature is used, otherwise no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs and JavaDocs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kbialek/flink FLINK-8660
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5530.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5530
----
commit 494bb2f7175bb174ed2a7139495af05891012498
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-17T15:11:50Z
Implemented
commit 077102571ae58987fafbf00ba536f1512c700e1b
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-17T16:45:47Z
Create BlobStorage for any HA backend
commit 57e52b70700dc1b3a4f05113ae6aa594112eead7
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-17T16:46:02Z
HighAvailabilityServicesFactory may throw exceptions
commit 9a08884ee05f8a8bf8e03b7fdd3b6d148530c40b
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-19T21:12:59Z
Docs
----
---
[GitHub] flink issue #5530: [FLINK-8660] Enable the user to provide custom HAServices...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5530
I think this looks quite good now.
Since 1.5 feature freeze is over for a bit, would suggest to merge this into master (1.6-SNAPSHOT).
Looking at the code (this PR and the high availability configuration in general), I would suggest to work towards the following design in the future:
- We don't have specific high availability enums and setting, everything is just an implementation of HAServices. Some HAServices are in fact not highly available (maybe we need to find a better name for HAServices).
- There are factories for all HAServices
- We simply have "none" and "zookeeper" as aliases for the respective factories.
That mirrors the design of he state backends and their configuration, which is nice, because we never need to do special checks like "isHighAvailabilityEnabled".
---
[GitHub] flink issue #5530: [FLINK-8660] Enable the user to provide custom HAServices...
Posted by kbialek <gi...@git.apache.org>.
Github user kbialek commented on the issue:
https://github.com/apache/flink/pull/5530
Closing without merge, because of:
1. Test failure
2. I'd like to implement Stephan's idea
---
[GitHub] flink issue #5530: [FLINK-8660] Enable the user to provide custom HAServices...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/5530
You could rebase on the latest master and force push to retrigger the build.
---
[GitHub] flink issue #5530: [FLINK-8660] Enable the user to provide custom HAServices...
Posted by kbialek <gi...@git.apache.org>.
Github user kbialek commented on the issue:
https://github.com/apache/flink/pull/5530
Now uses high-availability configuration property can be set either to none, zookeeper or FQN of factory class
---
[GitHub] flink issue #5530: [FLINK-8660] Enable the user to provide custom HAServices...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5530
I think this is a nice approach.
I would suggest to not split the config options between `high-availability` and factory that is only used in *CUSTOM* mode, but use the `high-availability` option for both. The option describes the high availability services factory, with *NONE* and *ZOOKEEPER* as special constants/shortcuts.
Have a look at the `StateBackendLoader` class for an example of that.
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L92
---
[GitHub] flink pull request #5530: [FLINK-8660] Enable the user to provide custom HAS...
Posted by kbialek <gi...@git.apache.org>.
Github user kbialek closed the pull request at:
https://github.com/apache/flink/pull/5530
---
[GitHub] flink pull request #5530: [FLINK-8660] Enable the user to provide custom HAS...
Posted by kbialek <gi...@git.apache.org>.
GitHub user kbialek reopened a pull request:
https://github.com/apache/flink/pull/5530
[FLINK-8660] Enable the user to provide custom HAServices implementation
## What is the purpose of the change
This pull request makes it possible to use alternative (to ZK) HA backend implementation
## Brief change log
* Added HighAvailabilityServicesFactory interface
* Added new option HighAvailabilityMode.CUSTOM
* Added new configuration property high-availability.factoryClass
* HighAvailabilityServicesUtils uses configured factory to instantiate HighAvailabilityServices
* Updated docs
## Verifying this change
This change added tests and can be verified as follows:
* Added new test class HighAvailabilityServicesUtilsTest
* Updated test class HighAvailabilityModeTest
* Manually verified by using own HA backend implementation
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes, if this feature is used, otherwise no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs and JavaDocs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kbialek/flink FLINK-8660
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5530.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5530
----
commit 494bb2f7175bb174ed2a7139495af05891012498
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-17T15:11:50Z
Implemented
commit 077102571ae58987fafbf00ba536f1512c700e1b
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-17T16:45:47Z
Create BlobStorage for any HA backend
commit 57e52b70700dc1b3a4f05113ae6aa594112eead7
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-17T16:46:02Z
HighAvailabilityServicesFactory may throw exceptions
commit 9a08884ee05f8a8bf8e03b7fdd3b6d148530c40b
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-19T21:12:59Z
Docs
commit 0afc341077347e32669330ea66d8518d9a3133ef
Author: zentol <ch...@...>
Date: 2018-02-07T08:52:23Z
[FLINK-8574][travis] Add timestamp to logging messages
This closes #5419.
commit e2f1650ea251ab7a7f6090cda819e2e2481ffdf5
Author: zentol <ch...@...>
Date: 2018-02-13T09:31:31Z
[FLINK-8621][prometheus][tests] Remove endpointIsUnavailableAfterReporterIsClosed()
The test is inherently unstable as it will always fail if any other
server is started on the port between the closing of the reporter and
the polling of metrics.
This closes #5473.
commit 2db296973ec2aa648246279a2e55cca87432f829
Author: zentol <ch...@...>
Date: 2018-02-13T09:33:03Z
[hotfix][prometheus] Document internal usage of CollectorRegistry.defaultRegistry
It appeared as if the HTTPServer wasn't actually doing anything, but it
internally accessed the singleton registry that we also access to
register metrics.
commit 6fb2c89f7cfa5b3d05c683777b76d257db4770c5
Author: zentol <ch...@...>
Date: 2018-02-13T09:33:31Z
[hotfix][prometheus][tests] Add utility for generating port ranges
commit 7441b16a4eeb10633aca4b9c89629ee92a182092
Author: Aljoscha Krettek <al...@...>
Date: 2018-02-14T13:48:22Z
[FLINK-8600] Allow disabling truncate() check in BucketingSink
The test was failing when using PrestoS3FileSystem because it doesn't
use an absolute/qualified path.
commit 9f5e835e845901fd1183b390c510851c2ca17b03
Author: Till Rohrmann <tr...@...>
Date: 2018-02-19T11:00:08Z
[FLINK-8698] [flip6] Use Flip6LocalStreamEnvironment instead of LocalStreamEnvironment
commit 9b135941c5295af4db074aa50d5bb881a6a52feb
Author: Till Rohrmann <tr...@...>
Date: 2018-02-19T11:41:41Z
[FLINK-8698] [flip6] Let LocalExecutor use Flip-6 MiniCluster
This closes #5524.
commit 4596c656c79829e2c73aee252ce13c0cb5bc9b7a
Author: Till Rohrmann <tr...@...>
Date: 2018-02-20T08:00:31Z
[FLINK-8709] [tests] Harden SlotPoolRpcTest.testCancelSlotAllocationWithoutResourceManager
commit 2d9db7204d3c4b012600416038679941241f1088
Author: zentol <ch...@...>
Date: 2018-02-20T16:31:05Z
[hotfix][docs] Update configuration docs
commit f75b763a40dcb6a1b1b06a7312d5ee7b67f19a5c
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-21T21:44:30Z
Use ha mode config property to specify factory class FQN
commit a93ed1f2f5045a7473788992db94235d8f6ec502
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-21T21:46:31Z
Merge branch 'master' into FLINK-8660
commit c97de5d4caf8eb99835003b624c4498221f4d76f
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-22T06:17:27Z
Naming refactor
commit 19658f3e428abe3432214e76f355dac4b0bd657e
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-22T06:18:30Z
Merge branch 'master' into FLINK-8660
commit 086c42bfa08c55a2def0520212d331c6a7387ff4
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-23T20:56:13Z
Merge branch 'master' into FLINK-8660
commit c89d064e83d504e26deeceaf5a1d4a6f98a0c3e9
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-23T21:36:29Z
Modify .travis.yml to trigger build
commit d6a8db73b22e465292e76769f51d40ee3f9ca113
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-23T21:39:06Z
Revert "Modify .travis.yml to trigger build"
This reverts commit c89d064
commit cf2f49fe33a7a24a9a11ebd419d7765f41d9c0af
Author: Krzysztof Białek <kb...@...>
Date: 2018-02-24T13:31:59Z
Update docs
----
---
[GitHub] flink issue #5530: [FLINK-8660] Enable the user to provide custom HAServices...
Posted by kbialek <gi...@git.apache.org>.
Github user kbialek commented on the issue:
https://github.com/apache/flink/pull/5530
Looks like build process was terminated by kernel. Can someone retrigger it?
---