You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/07/23 13:11:27 UTC
[GitHub] [flink] yittg opened a new pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
yittg opened a new pull request #16583:
URL: https://github.com/apache/flink/pull/16583
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Guarantee the watching event handling order manually in KubernetesSharedInformer instead of depending on a lock.
## Brief change log
## Verifying this change
This change is already covered by existing tests, such as `KubernetesSharedInformerITCase`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* eb5d06214f3f898d250d9bb3ff9b8a41a46700e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900)
* 9687f4fde6c3006d2fba5185eadc60c53abdddc2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* eb5d06214f3f898d250d9bb3ff9b8a41a46700e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921",
"triggerID" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20930",
"triggerID" : "886314172",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20930)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886205228
It looks really good now. I will get it merged tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
dmvk commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675579447
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
nit: We may want to use a blocking poll here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921",
"triggerID" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 48ca3133cc8d6c2b5505217decff95ed11a84847 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910)
* d889ba3c168d1b46ec7784e357f7350df86de86a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919)
* 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886131221
@yittg Maybe I do not make myself clear. What I mean is to do the manual verification via `jstack` in the JobManager pod to check the thread num of cached thread pool `config-map-watch-handler`. We need to build a customized image which is based on `openjdk:8-jdk`, not the `openjdk:8-jre`.
I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 9687f4fde6c3006d2fba5185eadc60c53abdddc2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901)
* 48ca3133cc8d6c2b5505217decff95ed11a84847 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886199402
> @wangyang0918 I've split this PR into two commit.
Thanks.
nit: Usually, we have the `hotfix` commit in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675926561
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,17 +265,17 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
- () -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
- }
- });
+ return;
}
+ Preconditions.checkState(
+ callbackQueue.add(handlerConsumer), "Unable to put callback into a queue.");
+ executorService.execute(
+ () -> {
+ synchronized (callbackLock) {
+ Objects.requireNonNull(callbackQueue.poll(), "Callback queue is empty")
Review comment:
```suggestion
Preconditions.checkNotNull(callbackQueue.poll(), "Callback queue is empty")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885753989
Thanks @wangyang0918 @dmvk ;
So here are three options if i understand correctly?
1. Revert the previous merged change;
2. Fix with simple lock solution provided by @dmvk; which i think will lead to many thread if events swarm but it is expected not happen normally. And another confuse point is the submit runnable will handle event that may not be corresponding with the current event.
3. Fix with the current PR , i have to admit that it need some thinking to read it.
So would you vote it? WDYT @tillrohrmann
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 closed pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 closed pull request #16583:
URL: https://github.com/apache/flink/pull/16583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885630061
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 6d52dc0b2a06b3aab47330cded257aac8ba306f8 (Fri Jul 23 13:14:15 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d52dc0b2a06b3aab47330cded257aac8ba306f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888)
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921",
"triggerID" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* d889ba3c168d1b46ec7784e357f7350df86de86a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919)
* 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675693787
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +266,32 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
Review comment:
We need to use the `execute` instead of `submit` here. Right? Because it can swallow exceptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d52dc0b2a06b3aab47330cded257aac8ba306f8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675588061
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
no need blocking poll, the poll can always get a element.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d52dc0b2a06b3aab47330cded257aac8ba306f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888)
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675606956
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
Yeah, it's also ok. i think it still needs some guard to make the single thread enter and exit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* e3cfa005beefb341b54371390694d81710879cc7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899)
* eb5d06214f3f898d250d9bb3ff9b8a41a46700e7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* eb5d06214f3f898d250d9bb3ff9b8a41a46700e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900)
* 9687f4fde6c3006d2fba5185eadc60c53abdddc2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886140470
I have verified in a HA-enabled session cluster with 20 running jobs, the thread num of `config-map-watch-handler` is always smaller than 7, which means we could save the threads and network connections via using a shared cached thread pool.
I have also run the tests in `KubernetesSharedInformerITCase` more than 100 times without failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* e3cfa005beefb341b54371390694d81710879cc7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
dmvk commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675594854
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
I think there is a rare case where it won't, basically if you poll between `handling.getAndIncrement() > 0` and
`queue.put(handlerConsumer);`. Also if `queue.put(handlerConsumer)` would get interrupted in case `present` is larger than `0` it may result in infinite busy waiting loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675589947
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
I've verify locally for many rounds, no exception ever found so far.
The execution thread will only survive for a short period only if new event comes before the last event complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* e3cfa005beefb341b54371390694d81710879cc7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899)
* eb5d06214f3f898d250d9bb3ff9b8a41a46700e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
dmvk commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675667186
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
Personally i find it harder to think and reason about as incrementing number of elements in the queue and actually putting and element into the queue is not atomic.
I'd personally prefer something like:
```
Preconditions.checkState(
callbackQueue.add(handlerConsumer), "Unable to put callback into a queue.");
executorService.submit(
() -> {
callbackLock.lock();
try {
Objects.requireNonNull(
callbackQueue.poll(), "Callback queue is empty.")
.accept(handler);
} finally {
callbackLock.unlock();
}
});
```
But that being said, I don't think I have a strong objection against your solution. WDYT @wangyang0918 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886007592
> Could you please verify that how many threads(in the cached thread pool for watching) we will have when running a HA enabled Flink application?
@wangyang0918 I'm not quit sure where is the proper place to do the verification. Is 'test_kubernetes_application_ha.sh' the right place? Do you have some good idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886199402
> @wangyang0918 I've split this PR into two commit.
@yittg Thanks.
nit: Usually, we have the `hotfix` commit in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675590385
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
i mean the loop by "The execution thread will only survive"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d52dc0b2a06b3aab47330cded257aac8ba306f8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888)
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675588061
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
no need blocking poll, the poll can always get an element.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886131221
@yittg Maybe I do not make myself clear. What I means is to do the manual verification via `jstack` in the JobManager pod to check the thread num of cached thread pool `config-map-watch-handler`. We need to build a customized image which is based on `openjdk:8-jdk`, not the `openjdk:8-jre`.
I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885628185
cc @wangyang0918 @dmvk @tillrohrmann
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d52dc0b2a06b3aab47330cded257aac8ba306f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885950936
> I would prefer the option 4.
>
> Passing a dedicated single thread, instead of the current shared cached thread pool, when call the `KubernetesSharedInformer#watch`.
@wangyang0918 Did you mean change the executor service initialized in `KubernetesHaServices`? @dmvk and i discussed before about leaving the order guarantee on the caller side. But we think it should be guaranteed no matter what executor the caller uses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 48ca3133cc8d6c2b5505217decff95ed11a84847 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675898651
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
@dmvk After more thinking, I now prefer your solution. If the caller uses a shared single-threaded executor to call watch, the current implementation may cause the thread to always be occupied by a single watcher. Your solution can avoid this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
dmvk commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675599266
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
> i think execute callbacks one by one and a new Runnable for each one needs another thread to maintain the order also.
BlockingQueue where you store incoming callbacks from a single thread (this maintains order) + lock (doesn't need to be fair, ordering is maintained by the queue, we just need an exclusive access), inside which you poll callback from the queue and execute it, should be enough.
---
Problem with the previous implementation was, that poll from the executor service queue, was not guarded by the lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886204761
haha, we pushed the update almost at the same time :), PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886168995
@yittg The last one comment. We need to factor out the `executorService.submit` -> `executorService.execute` related changes into a separate `hotfix` commit. Once this is done, I will merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
dmvk commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675576934
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
This makes sense 👍 Have you been able to verify locally that this fixes the issue?
I'm just thinking whether executing callbacks in a loop is ok (does it have potential to block the thread for a long time?), or we should execute callbacks one by one (each callback gets a new Runnable). IMO it's ok as is, just a first thought I had when I was looking at the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 48ca3133cc8d6c2b5505217decff95ed11a84847 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910)
* d889ba3c168d1b46ec7784e357f7350df86de86a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675606956
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
Yeah, it's also ok. i think it still some guard to make the single thread enter and exit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885758095
> Thanks @wangyang0918 @dmvk ;
> So here are three options if i understand correctly?
>
> 1. Revert the previous merged change;
> 2. Fix with simple lock solution provided by @dmvk; which i think will lead to many thread if events swarm but it is expected not happen normally. And another confuse point is the submit runnable will handle event that may not be corresponding with the current event.
> 3. Fix with the current PR , i have to admit that it need some thinking to read it.
>
> So would you vote it? WDYT @tillrohrmann
I would prefer the option 4.
Passing a dedicated single thread, instead of the current shared cached thread pool, when call the `KubernetesSharedInformer#watch`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885742344
> > I have to admit that using a dedicated thread for each watch will make the code easier to read and maintain. IIRC, each job will need three threads for watching the HA related ConfigMaps. It is reasonable and does not take too much overhead.
>
> Which solution are you referring to?
Given that our original purpose of FLINK-22054 is to save the network connections to K8s APIServer.
What I mean is passing a dedicated single thread when call the `KubernetesSharedInformer#watch`. The current implementation could fix the race condition, however it really makes the code not very easy to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921",
"triggerID" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20930",
"triggerID" : "886314172",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20930)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675641227
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
So, @dmvk would you mind i keep the current implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885725731
@yittg @dmvk I have to admit that using a dedicated thread for each watch will make the code easier to read and maintain. IIRC, each job will need three threads for watching the HA related ConfigMaps. It is reasonable and does not take too much overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 9687f4fde6c3006d2fba5185eadc60c53abdddc2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901)
* 48ca3133cc8d6c2b5505217decff95ed11a84847 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921",
"triggerID" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20921)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885630061
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 (Sat Aug 28 12:21:45 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886403407
Thanks @wangyang0918 , @dmvk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 9687f4fde6c3006d2fba5185eadc60c53abdddc2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886140470
I have verified in a HA-enabled session cluster with 20 running jobs, the thread num of `config-map-watch-handler` is always smaller than 6, which means we could save the threads and network connections via using a shared cached thread pool.
I have also run the tests in `KubernetesSharedInformerITCase` more than 100 times without failure.
I will merge this PR once the azure pipeline gives green paas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11dc520b8de778ba970d39914d1e7b9ffb5b9872 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891)
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* e3cfa005beefb341b54371390694d81710879cc7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899)
* eb5d06214f3f898d250d9bb3ff9b8a41a46700e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886195943
@wangyang0918 I've split this PR into two commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
}, {
"hash" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 48ca3133cc8d6c2b5505217decff95ed11a84847 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910)
* d889ba3c168d1b46ec7784e357f7350df86de86a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919)
* 93fbe85c4b7ff0faa3c3bcdd5a51a116349ec550 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on a change in pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on a change in pull request #16583:
URL: https://github.com/apache/flink/pull/16583#discussion_r675595456
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesSharedInformer.java
##########
@@ -265,14 +265,26 @@ private WatchCallback(
private void run(Consumer<WatchCallbackHandler<T>> handlerConsumer) {
if (this.executorService == null) {
handlerConsumer.accept(handler);
- } else {
- this.executorService.submit(
+ return;
+ }
+ final int present = handling.getAndIncrement();
+ try {
+ queue.put(handlerConsumer);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ }
+ if (present == 0) {
+ executorService.submit(
() -> {
- callbackLock.lock();
- try {
- handlerConsumer.accept(handler);
- } finally {
- callbackLock.unlock();
+ while (true) {
+ final Consumer<WatchCallbackHandler<T>> h = queue.poll();
Review comment:
i think execute callbacks one by one and a new Runnable for each one needs another thread to maintain the order also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dmvk commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
dmvk commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885737142
> I have to admit that using a dedicated thread for each watch will make the code easier to read and maintain. IIRC, each job will need three threads for watching the HA related ConfigMaps. It is reasonable and does not take too much overhead.
Which solution are you referring to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] yittg commented on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
yittg commented on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886314172
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-886140470
I have verified in a HA-enabled session cluster with 20 running jobs, the thread num of `config-map-watch-handler` is always smaller than 7, which means we could save the threads and network connections via using a shared cached thread pool.
I have also run the tests in `KubernetesSharedInformerITCase` more than 100 times without failure.
I will merge this PR once the azure pipeline gives green paas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16583: [FLINK-23478][k8s] Guarantee the watching event handling order
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16583:
URL: https://github.com/apache/flink/pull/16583#issuecomment-885655624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20888",
"triggerID" : "6d52dc0b2a06b3aab47330cded257aac8ba306f8",
"triggerType" : "PUSH"
}, {
"hash" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20891",
"triggerID" : "11dc520b8de778ba970d39914d1e7b9ffb5b9872",
"triggerType" : "PUSH"
}, {
"hash" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81a5e20202b5e8e50c97bfe8d39296f03af6ca4d",
"triggerType" : "PUSH"
}, {
"hash" : "e3cfa005beefb341b54371390694d81710879cc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20899",
"triggerID" : "e3cfa005beefb341b54371390694d81710879cc7",
"triggerType" : "PUSH"
}, {
"hash" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20900",
"triggerID" : "eb5d06214f3f898d250d9bb3ff9b8a41a46700e7",
"triggerType" : "PUSH"
}, {
"hash" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20901",
"triggerID" : "9687f4fde6c3006d2fba5185eadc60c53abdddc2",
"triggerType" : "PUSH"
}, {
"hash" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910",
"triggerID" : "48ca3133cc8d6c2b5505217decff95ed11a84847",
"triggerType" : "PUSH"
}, {
"hash" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919",
"triggerID" : "d889ba3c168d1b46ec7784e357f7350df86de86a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81a5e20202b5e8e50c97bfe8d39296f03af6ca4d UNKNOWN
* 48ca3133cc8d6c2b5505217decff95ed11a84847 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20910)
* d889ba3c168d1b46ec7784e357f7350df86de86a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20919)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org