You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2017/08/01 07:42:00 UTC
[GitHub] flink pull request #4444: [FLINK-7331] [futures] Replace Flink's Future with...
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/4444
[FLINK-7331] [futures] Replace Flink's Future with Java 8's CompletableFuture in ResourceManager
## What is the purpose of the change
Replace Flink's Future with Java 8's CompletableFuture in ResourceManager.
This PR is based on #4443.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink rfResourceManager
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4444.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4444
----
commit afe1d171132bb3724e672a1c4ce74a3f7c185908
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-31T13:07:18Z
[FLINK-7313] [futures] Add Flink future and Scala future to Java 8 CompletableFuture conversion
Add DirectExecutionContext
Add Scala Future to Java 8 CompletableFuture utility to FutureUtils
Add Flink future to Java 8's CompletableFuture conversion utility to FutureUtils
Add base class for Flink's unchecked future exceptions
commit 8425e85f3ea0499fb59fc3c25c035a622e3b282d
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-31T16:47:22Z
[FLINK-7321] [futures] Replace Flink's futures with Java 8's CompletableFuture in HeartbeatManager
commit 5b7dda9c974d2aa49648732efd5e77f8a9705432
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-31T17:35:14Z
[FLINK-7324] [futures] Replace Flink's future with Java 8's CompletableFuture in SlotPool
commit 8bc3dc9254c4bcfcf003f32667f8c05a371b4d3f
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-31T19:38:28Z
[FLINK-7328] [futures] Replace Flink's futures with Java 8's CompletableFuture in SlotManager
commit 14ec0af951b13e82390ffbee89597bd8c4414bf4
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-31T19:47:43Z
[FLINK-7331] [futures] Replace Flink's Future with Java 8's CompletableFuture in ResourceManager
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4444: [FLINK-7331] [futures] Replace Flink's Future with...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4444
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4444: [FLINK-7331] [futures] Replace Flink's Future with...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4444#discussion_r130549382
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java ---
@@ -401,74 +354,37 @@ public RegistrationResponse apply(RegistrationResponse registrationResponse, Thr
* @return The response by the ResourceManager.
*/
@RpcMethod
- public Future<RegistrationResponse> registerTaskExecutor(
--- End diff --
Unintended renaming? This method is annotated with `@RcpMethod`, we need to change `ResourceManagerGateway` as well. Although this begs the question why we expose an _internal_ method in the first place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4444: [FLINK-7331] [futures] Replace Flink's Future with...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4444#discussion_r130548682
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java ---
@@ -653,6 +565,130 @@ UUID getLeaderSessionId() {
// Internal methods
// ------------------------------------------------------------------------
+ /**
+ * Registers a new JobMaster.
+ *
+ * @param jobMasterGateway to communicate with the registering JobMaster
+ * @param jobLeaderId leader id of the JobMaster
+ * @param jobId of the job for which the JobMaster is responsible
+ * @param jobManagerAddress address of the JobMaster
+ * @param jobManagerResourceId ResourceID of the JobMaster
+ * @return RegistrationResponse
+ */
+ private RegistrationResponse registerJobMasterInternal(
+ final JobMasterGateway jobMasterGateway,
+ UUID jobLeaderId,
+ JobID jobId,
+ String jobManagerAddress,
+ ResourceID jobManagerResourceId) {
+ if (jobManagerRegistrations.containsKey(jobId)) {
+ JobManagerRegistration oldJobManagerRegistration = jobManagerRegistrations.get(jobId);
+
+ if (oldJobManagerRegistration.getLeaderID().equals(jobLeaderId)) {
+ // same registration
+ log.debug("Job manager {}@{} was already registered.", jobLeaderId, jobManagerAddress);
+ } else {
+ // tell old job manager that he is no longer the job leader
+ disconnectJobManager(
+ oldJobManagerRegistration.getJobID(),
+ new Exception("New job leader for job " + jobId + " found."));
+
+ JobManagerRegistration jobManagerRegistration = new JobManagerRegistration(
+ jobId,
+ jobManagerResourceId,
+ jobLeaderId,
+ jobMasterGateway);
+ jobManagerRegistrations.put(jobId, jobManagerRegistration);
+ jmResourceIdRegistrations.put(jobManagerResourceId, jobManagerRegistration);
+ }
+ } else {
+ // new registration for the job
+ JobManagerRegistration jobManagerRegistration = new JobManagerRegistration(
+ jobId,
+ jobManagerResourceId,
+ jobLeaderId,
+ jobMasterGateway);
+ jobManagerRegistrations.put(jobId, jobManagerRegistration);
+ jmResourceIdRegistrations.put(jobManagerResourceId, jobManagerRegistration);
+ }
+
+ log.info("Registered job manager {}@{} for job {}.", jobLeaderId, jobManagerAddress, jobId);
+
+ jobManagerHeartbeatManager.monitorTarget(jobManagerResourceId, new HeartbeatTarget<Void>() {
+ @Override
+ public void receiveHeartbeat(ResourceID resourceID, Void payload) {
+ // the ResourceManager will always send heartbeat requests to the JobManager
+ }
+
+ @Override
+ public void requestHeartbeat(ResourceID resourceID, Void payload) {
+ jobMasterGateway.heartbeatFromResourceManager(resourceID);
+ }
+ });
+
+ return new JobMasterRegistrationSuccess(
+ resourceManagerConfiguration.getHeartbeatInterval().toMilliseconds(),
+ getLeaderSessionId(),
+ resourceId);
+ }
+
+ /**
+ * Registers a new TaskExecutor.
+ *
+ * @param taskExecutorGateway to communicate with the registering TaskExecutor
+ * @param taskExecutorAddress address of the TaskExecutor
+ * @param taskExecutorResourceId ResourceID of the TaskExecutor
+ * @param slotReport initial slot report from the TaskExecutor
+ * @return RegistrationResponse
+ */
+ private RegistrationResponse registerTaskExecutorInternal(
+ TaskExecutorGateway taskExecutorGateway,
+ String taskExecutorAddress,
+ ResourceID taskExecutorResourceId,
+ SlotReport slotReport) {
+ WorkerRegistration<WorkerType> oldRegistration = taskExecutors.remove(taskExecutorResourceId);
+ if (oldRegistration != null) {
+ // TODO :: suggest old taskExecutor to stop itself
+ log.info("Replacing old instance of worker for ResourceID {}", taskExecutorResourceId);
+
+ // remove old task manager registration from slot manager
+ slotManager.unregisterTaskManager(oldRegistration.getInstanceID());
+ }
+
+ final WorkerType newWorker = workerStarted(taskExecutorResourceId);
+
+ if(newWorker == null) {
--- End diff --
add space after if
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4444: [FLINK-7331] [futures] Replace Flink's Future with...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4444#discussion_r130592145
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java ---
@@ -653,6 +565,130 @@ UUID getLeaderSessionId() {
// Internal methods
// ------------------------------------------------------------------------
+ /**
+ * Registers a new JobMaster.
+ *
+ * @param jobMasterGateway to communicate with the registering JobMaster
+ * @param jobLeaderId leader id of the JobMaster
+ * @param jobId of the job for which the JobMaster is responsible
+ * @param jobManagerAddress address of the JobMaster
+ * @param jobManagerResourceId ResourceID of the JobMaster
+ * @return RegistrationResponse
+ */
+ private RegistrationResponse registerJobMasterInternal(
+ final JobMasterGateway jobMasterGateway,
+ UUID jobLeaderId,
+ JobID jobId,
+ String jobManagerAddress,
+ ResourceID jobManagerResourceId) {
+ if (jobManagerRegistrations.containsKey(jobId)) {
+ JobManagerRegistration oldJobManagerRegistration = jobManagerRegistrations.get(jobId);
+
+ if (oldJobManagerRegistration.getLeaderID().equals(jobLeaderId)) {
+ // same registration
+ log.debug("Job manager {}@{} was already registered.", jobLeaderId, jobManagerAddress);
+ } else {
+ // tell old job manager that he is no longer the job leader
+ disconnectJobManager(
+ oldJobManagerRegistration.getJobID(),
+ new Exception("New job leader for job " + jobId + " found."));
+
+ JobManagerRegistration jobManagerRegistration = new JobManagerRegistration(
+ jobId,
+ jobManagerResourceId,
+ jobLeaderId,
+ jobMasterGateway);
+ jobManagerRegistrations.put(jobId, jobManagerRegistration);
+ jmResourceIdRegistrations.put(jobManagerResourceId, jobManagerRegistration);
+ }
+ } else {
+ // new registration for the job
+ JobManagerRegistration jobManagerRegistration = new JobManagerRegistration(
+ jobId,
+ jobManagerResourceId,
+ jobLeaderId,
+ jobMasterGateway);
+ jobManagerRegistrations.put(jobId, jobManagerRegistration);
+ jmResourceIdRegistrations.put(jobManagerResourceId, jobManagerRegistration);
+ }
+
+ log.info("Registered job manager {}@{} for job {}.", jobLeaderId, jobManagerAddress, jobId);
+
+ jobManagerHeartbeatManager.monitorTarget(jobManagerResourceId, new HeartbeatTarget<Void>() {
+ @Override
+ public void receiveHeartbeat(ResourceID resourceID, Void payload) {
+ // the ResourceManager will always send heartbeat requests to the JobManager
+ }
+
+ @Override
+ public void requestHeartbeat(ResourceID resourceID, Void payload) {
+ jobMasterGateway.heartbeatFromResourceManager(resourceID);
+ }
+ });
+
+ return new JobMasterRegistrationSuccess(
+ resourceManagerConfiguration.getHeartbeatInterval().toMilliseconds(),
+ getLeaderSessionId(),
+ resourceId);
+ }
+
+ /**
+ * Registers a new TaskExecutor.
+ *
+ * @param taskExecutorGateway to communicate with the registering TaskExecutor
+ * @param taskExecutorAddress address of the TaskExecutor
+ * @param taskExecutorResourceId ResourceID of the TaskExecutor
+ * @param slotReport initial slot report from the TaskExecutor
+ * @return RegistrationResponse
+ */
+ private RegistrationResponse registerTaskExecutorInternal(
+ TaskExecutorGateway taskExecutorGateway,
+ String taskExecutorAddress,
+ ResourceID taskExecutorResourceId,
+ SlotReport slotReport) {
+ WorkerRegistration<WorkerType> oldRegistration = taskExecutors.remove(taskExecutorResourceId);
+ if (oldRegistration != null) {
+ // TODO :: suggest old taskExecutor to stop itself
+ log.info("Replacing old instance of worker for ResourceID {}", taskExecutorResourceId);
+
+ // remove old task manager registration from slot manager
+ slotManager.unregisterTaskManager(oldRegistration.getInstanceID());
+ }
+
+ final WorkerType newWorker = workerStarted(taskExecutorResourceId);
+
+ if(newWorker == null) {
--- End diff --
Good catch. Will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4444: [FLINK-7331] [futures] Replace Flink's Future with Java 8...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4444
Thanks for the review @zentol. I addressed your comments. Merging this PR now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---