You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/03/27 17:12:45 UTC
[GitHub] [flink] tillrohrmann opened a new pull request #11552:
[FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when
target loses leadership
tillrohrmann opened a new pull request #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552
## What is the purpose of the change
Clearing the rpcConnection field in the JobManagerLeaderListener when target loses leadership
prevents that we try to reconnect to the target in case JobLeaderService.reconnect(JobID) is
called.
## Verifying this change
Added `JobLeaderServiceTest#doesNotReconnectAfterTargetLostLeadership`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11552: [FLINK-16836]
Clear rpcConnection field in JobManagerLeaderListener when target loses
leadership
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#issuecomment-605141925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6764",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
}, {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155913515",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155913515) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6764)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #11552:
[FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when
target loses leadership
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#discussion_r400719913
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/JobLeaderServiceTest.java
##########
@@ -96,21 +100,77 @@ public void go() throws Exception {
addJobAction.sync();
}
+ /**
+ * Tests that the JobLeaderService won't try to reconnect to JobMaster after it
+ * has lost the leadership. See FLINK-16836.
+ */
+ @Test
+ public void doesNotReconnectAfterTargetLostLeadership() throws Exception {
+ final JobLeaderService jobLeaderService = new JobLeaderService(
+ new LocalUnresolvedTaskManagerLocation(),
+ RetryingRegistrationConfiguration.defaultConfiguration());
+
+ final JobID jobId = new JobID();
+
+ final SettableLeaderRetrievalService leaderRetrievalService = new SettableLeaderRetrievalService();
+ final TestingHighAvailabilityServices haServices = new TestingHighAvailabilityServicesBuilder()
+ .setJobMasterLeaderRetrieverFunction(ignored -> leaderRetrievalService)
+ .build();
+
+ final String jmAddress = "foobar";
+ final TestingJobMasterGateway jobMasterGateway = new TestingJobMasterGatewayBuilder().build();
+ rpcServiceResource.getTestingRpcService().registerGateway(jmAddress, jobMasterGateway);
+
+ final TestingJobLeaderListener testingJobLeaderListener = new TestingJobLeaderListener();
+ jobLeaderService.start(
+ "foobar",
+ rpcServiceResource.getTestingRpcService(),
+ haServices,
+ testingJobLeaderListener);
+
+ try {
+ jobLeaderService.addJob(jobId, jmAddress);
+
+ leaderRetrievalService.notifyListener(jmAddress, UUID.randomUUID());
+
+ testingJobLeaderListener.waitUntilJobManagerGainedLeadership();
+
+ // revoke the leadership
+ leaderRetrievalService.notifyListener(null, null);
+ testingJobLeaderListener.waitUntilJobManagerLostLeadership();
+
+ jobLeaderService.reconnect(jobId);
+ } finally {
+ jobLeaderService.stop();
+ }
+ }
+
private static final class TestingJobLeaderListener implements JobLeaderListener {
+ private final CountDownLatch jobManagerGainedLeadership = new CountDownLatch(1);
+ private final CountDownLatch jobManagerLostLeadershp = new CountDownLatch(1);
+
@Override
public void jobManagerGainedLeadership(JobID jobId, JobMasterGateway jobManagerGateway, JMTMRegistrationSuccess registrationMessage) {
- // ignored
+ jobManagerGainedLeadership.countDown();
}
@Override
public void jobManagerLostLeadership(JobID jobId, JobMasterId jobMasterId) {
- // ignored
+ jobManagerLostLeadershp.countDown();
}
@Override
public void handleError(Throwable throwable) {
// ignored
}
+
+ private void waitUntilJobManagerGainedLeadership() throws InterruptedException {
+ jobManagerGainedLeadership.await();
+ }
+
+ private void waitUntilJobManagerLostLeadership() throws InterruptedException {
+ jobManagerLostLeadershp.await();
Review comment:
```suggestion
jobManagerLostLeadership.await();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #11552:
[FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when
target loses leadership
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#discussion_r400719813
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/JobLeaderServiceTest.java
##########
@@ -96,21 +100,77 @@ public void go() throws Exception {
addJobAction.sync();
}
+ /**
+ * Tests that the JobLeaderService won't try to reconnect to JobMaster after it
+ * has lost the leadership. See FLINK-16836.
+ */
+ @Test
+ public void doesNotReconnectAfterTargetLostLeadership() throws Exception {
+ final JobLeaderService jobLeaderService = new JobLeaderService(
+ new LocalUnresolvedTaskManagerLocation(),
+ RetryingRegistrationConfiguration.defaultConfiguration());
+
+ final JobID jobId = new JobID();
+
+ final SettableLeaderRetrievalService leaderRetrievalService = new SettableLeaderRetrievalService();
+ final TestingHighAvailabilityServices haServices = new TestingHighAvailabilityServicesBuilder()
+ .setJobMasterLeaderRetrieverFunction(ignored -> leaderRetrievalService)
+ .build();
+
+ final String jmAddress = "foobar";
+ final TestingJobMasterGateway jobMasterGateway = new TestingJobMasterGatewayBuilder().build();
+ rpcServiceResource.getTestingRpcService().registerGateway(jmAddress, jobMasterGateway);
+
+ final TestingJobLeaderListener testingJobLeaderListener = new TestingJobLeaderListener();
+ jobLeaderService.start(
+ "foobar",
+ rpcServiceResource.getTestingRpcService(),
+ haServices,
+ testingJobLeaderListener);
+
+ try {
+ jobLeaderService.addJob(jobId, jmAddress);
+
+ leaderRetrievalService.notifyListener(jmAddress, UUID.randomUUID());
+
+ testingJobLeaderListener.waitUntilJobManagerGainedLeadership();
+
+ // revoke the leadership
+ leaderRetrievalService.notifyListener(null, null);
+ testingJobLeaderListener.waitUntilJobManagerLostLeadership();
+
+ jobLeaderService.reconnect(jobId);
+ } finally {
+ jobLeaderService.stop();
+ }
+ }
+
private static final class TestingJobLeaderListener implements JobLeaderListener {
+ private final CountDownLatch jobManagerGainedLeadership = new CountDownLatch(1);
+ private final CountDownLatch jobManagerLostLeadershp = new CountDownLatch(1);
+
@Override
public void jobManagerGainedLeadership(JobID jobId, JobMasterGateway jobManagerGateway, JMTMRegistrationSuccess registrationMessage) {
- // ignored
+ jobManagerGainedLeadership.countDown();
}
@Override
public void jobManagerLostLeadership(JobID jobId, JobMasterId jobMasterId) {
- // ignored
+ jobManagerLostLeadershp.countDown();
Review comment:
```suggestion
jobManagerLostLeadership.countDown();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11552: [FLINK-16836]
Clear rpcConnection field in JobManagerLeaderListener when target loses
leadership
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#issuecomment-605141925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6764",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
}, {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155913515",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155913515) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6764)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11552: [FLINK-16836] Clear
rpcConnection field in JobManagerLeaderListener when target loses leadership
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#issuecomment-605124053
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8 (Fri Mar 27 17:14:44 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #11552: [FLINK-16836]
Clear rpcConnection field in JobManagerLeaderListener when target loses
leadership
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] GJL commented on a change in pull request #11552:
[FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when
target loses leadership
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#discussion_r400719697
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/JobLeaderServiceTest.java
##########
@@ -96,21 +100,77 @@ public void go() throws Exception {
addJobAction.sync();
}
+ /**
+ * Tests that the JobLeaderService won't try to reconnect to JobMaster after it
+ * has lost the leadership. See FLINK-16836.
+ */
+ @Test
+ public void doesNotReconnectAfterTargetLostLeadership() throws Exception {
+ final JobLeaderService jobLeaderService = new JobLeaderService(
+ new LocalUnresolvedTaskManagerLocation(),
+ RetryingRegistrationConfiguration.defaultConfiguration());
+
+ final JobID jobId = new JobID();
+
+ final SettableLeaderRetrievalService leaderRetrievalService = new SettableLeaderRetrievalService();
+ final TestingHighAvailabilityServices haServices = new TestingHighAvailabilityServicesBuilder()
+ .setJobMasterLeaderRetrieverFunction(ignored -> leaderRetrievalService)
+ .build();
+
+ final String jmAddress = "foobar";
+ final TestingJobMasterGateway jobMasterGateway = new TestingJobMasterGatewayBuilder().build();
+ rpcServiceResource.getTestingRpcService().registerGateway(jmAddress, jobMasterGateway);
+
+ final TestingJobLeaderListener testingJobLeaderListener = new TestingJobLeaderListener();
+ jobLeaderService.start(
+ "foobar",
+ rpcServiceResource.getTestingRpcService(),
+ haServices,
+ testingJobLeaderListener);
+
+ try {
+ jobLeaderService.addJob(jobId, jmAddress);
+
+ leaderRetrievalService.notifyListener(jmAddress, UUID.randomUUID());
+
+ testingJobLeaderListener.waitUntilJobManagerGainedLeadership();
+
+ // revoke the leadership
+ leaderRetrievalService.notifyListener(null, null);
+ testingJobLeaderListener.waitUntilJobManagerLostLeadership();
+
+ jobLeaderService.reconnect(jobId);
+ } finally {
+ jobLeaderService.stop();
+ }
+ }
+
private static final class TestingJobLeaderListener implements JobLeaderListener {
+ private final CountDownLatch jobManagerGainedLeadership = new CountDownLatch(1);
+ private final CountDownLatch jobManagerLostLeadershp = new CountDownLatch(1);
Review comment:
```suggestion
private final CountDownLatch jobManagerLostLeadership = new CountDownLatch(1);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #11552: [FLINK-16836] Clear
rpcConnection field in JobManagerLeaderListener when target loses leadership
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on issue #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#issuecomment-606548723
Thanks for the review @GJL. Merging this PR after addressing your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11552: [FLINK-16836] Clear
rpcConnection field in JobManagerLeaderListener when target loses leadership
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#issuecomment-605141925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11552: [FLINK-16836]
Clear rpcConnection field in JobManagerLeaderListener when target loses
leadership
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11552: [FLINK-16836] Clear rpcConnection field in JobManagerLeaderListener when target loses leadership
URL: https://github.com/apache/flink/pull/11552#issuecomment-605141925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6764",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
}, {
"hash" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155913515",
"triggerID" : "758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 758c24ca411ea4a5d16b5cbadbfe9a89fd2af6b8 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/155913515) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6764)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services