You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/10/13 17:56:32 UTC
[GitHub] [flink-kubernetes-operator] sap1ens opened a new pull request, #403: [FLINK-29633] Pass fromSavepoint argument
sap1ens opened a new pull request, #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403
## What is the purpose of the change
Passing `initialSavepointPath` from the JobSpec as a `fromSavepoint` argument in the standalone mode.
## Brief change log
- Updated `StandaloneKubernetesJobManagerParameters` to include savepoint path.
- Updated `CmdStandaloneJobManagerDecorator` to pass that as `fromSavepoint`.
## Verifying this change
- Modified `CmdStandaloneJobManagerDecoratorTest` to confirm the argument is passed properly.
- About to test this change on the staging environment... Will update the status.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
- Core observer or reconciler logic that is regularly executed: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1279444456
> > We should think of a way to test these in a reliable way. Testing the parameters is not enough as might not be understood by the JM as you pointed out.
> > Maybe we need some e2e tests for standalone mode that test these params properly
>
> I agree, e2e testing seems to be the only reliable way to verify that it works.
>
> I've checked the `e2e-tests` folder and it looks like it contains only fairly basic tests. To properly test this change we need to:
>
> 1. Start a stateful application
> 2. Take a savepoint
> 3. Stop it
> 4. Start another stateful application with a slightly different topology so `allowNonRestoredState` is required
> 5. Confirm it's restored properly
>
> Perhaps we can skip steps 1-3 if we're ok with keeping a savepoint file as a test resource. Also, step 4 is particularly tricky: is there a standard Flink example that we can use? Otherwise, we'd need to build a new app just for testing.
>
> This feel like a new test file.
>
> @gyfora thoughts?
I agree, it would be nice to have comprehensive tests that actually test the allowNonRestoredState logic. On the other hand if we simply enable that flag on existing tests at least we provide a regression test for this particular issue.
Because current test would have still failed before this fix if that flag is enabled for standalone.
I think we should do something simple for this ticket and open another jira for a complete test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996347785
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
I don't insist I think it's good like this for more visibility, I just wanted to make sure I fully understand the reason behind this particular change. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996346689
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
Yep, I think you're right: it seems like just fixing the `allowNonRestoredState` flag would be enough.
However, I think it's logical to provide the two options together via the command-line. They always come together (check `SavepointConfigOptions`, `SavepointRestoreSettings`, etc.) and I was really surprised when I didn't see the `fromSavepoint` argument when debugging.
But if you insist on making the smallest change in this PR then I'll revert the `fromSavepoint` part. Let me know!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996327486
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
What I meant with my other question is actually why we need this part here :)
Seems like before this change the savepoint restore logic worked correctly so the config is probably enough.
Wouldn't be enough to simply fix the allowNonRestoredState logic to not put true/false there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996347944
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
actually I think it might be enough to remove both `fromSavepoint` and `allowNonRestoredState` command line arguments and rely purely on the config. If that works I would actually lean toward that. It's much simpler and more bulletproof to just use the conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1279307886
> We should think of a way to test these in a reliable way. Testing the parameters is not enough as might not be understood by the JM as you pointed out.
>
> Maybe we need some e2e tests for standalone mode that test these params properly
I agree, e2e testing seems to be the only reliable way to verify that it works.
I've checked the `e2e-tests` folder and it looks like it contains only fairly basic tests. To properly test this change we need to:
1. Start a stateful application
2. Take a savepoint
3. Stop it
4. Start another stateful application with a slightly different topology so `allowNonRestoredState` is required
5. Confirm it's restored properly
Perhaps we can skip steps 1-3 if we're ok with keeping a savepoint file as a test resource. Also, step 4 is particularly tricky: is there a standard Flink example that we can use? Otherwise, we'd need to build a new app just for testing.
This feel like a new test file.
@gyfora thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1279825176
@sap1ens could you please open a backport PR against the `release-1.2` branch based on the merged commit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora merged pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora merged PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996327566
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
As we discussed on Slack, before this fix. Both `initialSavepoint` and savepoint upgrade logic worked correctly so there must not really be a need for the `--fromSavepoint` argument
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1278278263
I was surprised to see that this change didn't fix the issue on staging... After spending more time on debugging I found that the `allowNonRestoredState` flag actually doesn't require a boolean! And Flink's parser can parse it incorrectly when it's supplied.
With `--allowNonRestoredState true`:
<img width="868" alt="Screen Shot 2022-10-13 at 4 19 04 PM" src="https://user-images.githubusercontent.com/260702/195728451-98801e92-4ec7-4ffc-9307-7b34a41fc922.png">
With `--allowNonRestoredState`:
<img width="857" alt="Screen Shot 2022-10-13 at 4 19 17 PM" src="https://user-images.githubusercontent.com/260702/195728453-a4870992-e854-4766-9c0e-f156c88184d4.png">
Pushed https://github.com/apache/flink-kubernetes-operator/pull/403/commits/add45fc400511280c0cf8cdd97d4514e2375fa57 as a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996211925
##########
flink-kubernetes-standalone/src/test/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecoratorTest.java:
##########
@@ -85,7 +86,8 @@ public void testApplicationCommandAndArgsAdded() {
containsInAnyOrder(
CmdStandaloneJobManagerDecorator.APPLICATION_MODE_ARG,
"--allowNonRestoredState",
- "false",
+ "--fromSavepoint",
+ "/tmp/savepoint/path",
Review Comment:
Because I'm testing the logic I added in the `CmdStandaloneJobManagerDecorator` file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996348488
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
I thought about it too, but based on these lines:
- https://github.com/apache/flink/blob/master/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterEntryPoint.java#L95
- https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/SavepointRestoreSettings.java#L166-L168
it looks like if the flag is not present on the command-line, Flink will override the value from the config file to `false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1277992453
cc @usamj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1279035826
We should think of a way to test these in a reliable way. Testing the parameters is not enough as might not be understood by the JM as you pointed out.
Maybe we need some e2e tests for standalone mode that test these params properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1279862016
Done https://github.com/apache/flink-kubernetes-operator/pull/404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1278294021
Sorry, GitHub decided to close it automatically after I referenced it in an internal PR that was merged :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens commented on pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens commented on PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#issuecomment-1279584573
> I agree, it would be nice to have comprehensive tests that actually test the allowNonRestoredState logic. On the other hand if we simply enable that flag on existing tests at least we provide a regression test for this particular issue.
>
> Because current test would have still failed before this fix if that flag is enabled for standalone.
Not quite: `allowNonRestoredState` is pretty much ignored unless `fromSavepoint` is also provided. WIth or without the boolean: it doesn't matter. That's why I think we need a real savepoint recovery test, probably as a separate issue / PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996349881
##########
flink-kubernetes-standalone/src/main/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecorator.java:
##########
@@ -80,9 +80,14 @@ private List<String> getApplicationClusterArgs() {
}
Boolean allowNonRestoredState = kubernetesJobManagerParameters.getAllowNonRestoredState();
- if (allowNonRestoredState != null) {
+ if (allowNonRestoredState != null && allowNonRestoredState) {
args.add("--allowNonRestoredState");
- args.add(allowNonRestoredState.toString());
+ }
+
+ String savepointPath = kubernetesJobManagerParameters.getSavepointPath();
+ if (savepointPath != null) {
+ args.add("--fromSavepoint");
+ args.add(savepointPath);
Review Comment:
You are right, let’s keep it as is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] sap1ens closed pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
sap1ens closed pull request #403: [FLINK-29633] Pass fromSavepoint argument
URL: https://github.com/apache/flink-kubernetes-operator/pull/403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #403: [FLINK-29633] Pass fromSavepoint argument
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #403:
URL: https://github.com/apache/flink-kubernetes-operator/pull/403#discussion_r996141173
##########
flink-kubernetes-standalone/src/test/java/org/apache/flink/kubernetes/operator/kubeclient/decorators/CmdStandaloneJobManagerDecoratorTest.java:
##########
@@ -85,7 +86,8 @@ public void testApplicationCommandAndArgsAdded() {
containsInAnyOrder(
CmdStandaloneJobManagerDecorator.APPLICATION_MODE_ARG,
"--allowNonRestoredState",
- "false",
+ "--fromSavepoint",
+ "/tmp/savepoint/path",
Review Comment:
Why do we need to add this here? Seems like it worked before without it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org