You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/01/13 10:17:54 UTC
[GitHub] [flink] wangyang0918 opened a new pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
wangyang0918 opened a new pull request #14629:
URL: https://github.com/apache/flink/pull/14629
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Now the native K8s integration is almost production ready. We already have the common features for deploying a Flink cluster on K8s cluster natively. However, compared with the current supported features[1] in flink-on-k8s-operator and the feedback from users(ML and JIRAs), I am afraid that we still need some advanced features.
* Init containers[2]
* Sidecar containers[3]
* Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount[4]
* Set owner reference for the JobManager deployment[5]
* Support customizing of containers for native kubernetes setup[6]
* Pod Affinity and Anti-affinity
* Pod Security Context
* Pod Environment Variables Reference
So this PR tries to introduce a pod template[7] for all the use cases. Users could configure a local yaml file for JobManager/TaskManager. This will let Flink create the JobManager/TaskManager pods with the template first, not the empty. Of course, the Flink config options for Kubernetes[8] have high priority and could overwrite these values.
Pod template is is a typical pod yaml. You could set arbitrary fields that is supported by K8s directly. The following is a simple example, which add the init container, sidecar container and volume mount.
```
apiVersion: v1
kind: Pod
metadata:
name: pod-template
spec:
initContainers:
- name: artifacts-fetcher
image: busybox
imagePullPolicy: IfNotPresent
# Use wget or other tools to get user jars from remote storage
command: ['wget', 'https://flink-debug-yiqi.oss-cn-beijing.aliyuncs.com/StateMachineExample.jar', '-O', '/flink-artifact/myjob.jar']
volumeMounts:
- mountPath: /flink-artifact
name: flink-artifact
containers:
# Do not change the main container name
- name: flink-job-manager
volumeMounts:
- mountPath: /opt/flink/volumes/hostpath
name: flink-volume-hostpath
- mountPath: /opt/flink/usrlib
name: flink-artifact
- mountPath: /opt/flink/log
name: flink-logs
# Use sidecar container to push logs to remote storage or do some other debugging things
- name: sidecar-log-collector
image: busybox
imagePullPolicy: IfNotPresent
args:
- tail
- -F
- /flink-log/jobmanager.log
volumeMounts:
- mountPath: /flink-log
name: flink-logs
volumes:
- name: flink-volume-hostpath
hostPath:
path: /tmp
type: Directory
- name: flink-artifact
emptyDir: {}
- name: flink-logs
emptyDir: {}
```
[1]. https://github.com/GoogleCloudPlatform/flink-on-k8s-operator#features
[2]. https://issues.apache.org/jira/browse/FLINK-15641
[3]. https://issues.apache.org/jira/browse/FLINK-15871
[4]. https://issues.apache.org/jira/browse/FLINK-15649
[5]. https://issues.apache.org/jira/browse/FLINK-20359
[6]. https://issues.apache.org/jira/browse/FLINK-20324
[7]. https://issues.apache.org/jira/browse/FLINK-15656
[8]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes
## Brief change log
* Introduce the pod template
## Verifying this change
* All the new introduced functionalities are covered by new added unit test
* Manually testing
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (Will add a detailed documentation about how to use pod template in a following PR)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-766512509
I believe your suggestions make sense. For labels, annotations, node-selectors, tolerations and other similar features, Flink config option will add additional results to such fields. They could be combined easily. However, for other Flink internal generated fields(e.g. `namespace`, `podName`, `restartPolicy`, `mem`, `cpu`, etc.), the values from pod template will be always overwritten.
This should be declared clearly in the documentation with few tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b93e4c21b1edf4cf9c071d88b78ab4230fbc004 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982)
* 04a3a8996bc66bb2d953d5bee380da1a5eff8eac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581639223
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
+
+ private static final String ENTRY_POINT_CLASS =
+ KubernetesSessionClusterEntrypoint.class.getCanonicalName();
+ private static final int RESOURCE_MEMORY = 1456;
+
+ /**
+ * The result pod variables should be set via {@link #setResultPod()} by implementation class.
+ */
+ protected ObjectMeta resultPodMeta;
+
+ protected PodSpec resultPodSpec;
+
+ @Override
+ protected void setupFlinkConfig() {
+ super.setupFlinkConfig();
+ flinkConfig.set(KubernetesConfigOptionsInternal.ENTRY_POINT_CLASS, ENTRY_POINT_CLASS);
+
+ // Set fixed ports
+ flinkConfig.set(RestOptions.PORT, Constants.REST_PORT);
+ flinkConfig.set(BlobServerOptions.PORT, Integer.toString(Constants.BLOB_SERVER_PORT));
+ flinkConfig.setString(
+ TaskManagerOptions.RPC_PORT, String.valueOf(Constants.TASK_MANAGER_RPC_PORT));
+
+ flinkConfig.set(
+ TaskManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.ofMebiBytes(RESOURCE_MEMORY));
+ }
+
+ @Override
+ public final void onSetup() throws Exception {
+ setResultPod();
+ }
+
+ protected abstract void setResultPod() throws Exception;
+
+ @Test
+ public void testInitContainerFromPodTemplate() {
+ assertThat(resultPodSpec.getInitContainers().size(), is(1));
+ assertThat(
+ resultPodSpec.getInitContainers().get(0),
+ is(KubernetesPodTemplateTestUtils.getInitContainerInPodTemplate()));
+ }
+
+ @Test
+ public void testSideCarContainerFromPodTemplate() {
+ final Container sideCarContainer =
+ KubernetesPodTemplateTestUtils.getContainerWithName(
+ resultPodSpec,
+ KubernetesPodTemplateTestUtils.TESTING_SIDE_CAR_CONTAINER_NAME);
+ assertThat(
+ sideCarContainer,
+ is(KubernetesPodTemplateTestUtils.getSideCarContainerInPodTemplate()));
+ }
+
+ @Test
+ public void testVolumesFromPodTemplate() {
+ assertThat(
+ resultPodSpec.getVolumes(),
+ hasItems(KubernetesPodTemplateTestUtils.getVolumesInPodTemplate()));
+ }
+
+ @Test
+ public void testMainContainerVolumeMountsFromPodTemplate() {
+ final Container mainContainer =
+ KubernetesPodTemplateTestUtils.getContainerWithName(
+ resultPodSpec, Constants.MAIN_CONTAINER_NAME);
+ assertThat(
+ mainContainer.getVolumeMounts(),
+ hasItems(KubernetesPodTemplateTestUtils.getVolumeMountInPodTemplate()));
+ }
Review comment:
I will load the pod template in the base class and only the let the implementation class to decorate it in `getResultPod(FlinkPod podTemplate)`. Then one could know that how the result pod is decorated in the base class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765097339
@blublinsky Thanks for your testing on the preview version and sharing your thoughts.
> The priority between pod template and Flink config options
This is a very good and important concern. When I try to introduce the pod template, I always have a rule in my mind, "Flink config options have a high priority than pod template". If you could enable this feature(e.g. labels, annotations, tolerations, etc.) via config options, then you should do it.
Flink allows users to define the JobManager and TaskManager via pod template file with the non-common features, which are not supported by Flink [Kubernetes config options](https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes) directly. Please use [kubernetes.jobmanager.pod-template-file](https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes-jobmanager-pod-template-file)
or [kubernetes.taskmanager.pod-template-file](https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes-taskmanager-pod-template-file) to specify a local file that contains pod definition. It will be used to **initialize** the JobManager/TaskManager instead of an empty pod. Please note that some fields of the pod template will always be overridden by Flink. Refer to the following tables for the full list of pod fields that will be overwritten by Flink. _I am now working to provide such tables in this PR._
I know some guys may argue for that. In the pod template design, it is a supplement for some un-common features and provide enough flexibility to use new K8s features. Also I believe for Flink users, they prefer the config option if they could.
> Both this and owner manager PRs are still not merged. - do you have a time frame for merging/releasing them?
I am trying to refine this PR and also provide a thorough documentation so that users could easily use it. For owner reference PR, I already left some comments few days ago, could you please address them before merging?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5426ab123aef03d4710c0fea1237fa014684f372 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918)
* 145f891faab2190d83a5ea35ac86a025605b43bd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976)
* 6b93e4c21b1edf4cf9c071d88b78ab4230fbc004 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-774606334
cc @xintongsong @tillrohrmann Could you please help with reviewing this PR? The functionality has already be verified by at least three community users and our internal use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5426ab123aef03d4710c0fea1237fa014684f372 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918)
* 145f891faab2190d83a5ea35ac86a025605b43bd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584124798
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -49,64 +50,104 @@
public class InitJobManagerDecorator extends AbstractKubernetesStepDecorator {
private final KubernetesJobManagerParameters kubernetesJobManagerParameters;
+ private final Configuration flinkConfig;
public InitJobManagerDecorator(KubernetesJobManagerParameters kubernetesJobManagerParameters) {
this.kubernetesJobManagerParameters = checkNotNull(kubernetesJobManagerParameters);
+ this.flinkConfig = checkNotNull(kubernetesJobManagerParameters.getFlinkConfiguration());
}
@Override
public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
- final Pod basicPod =
- new PodBuilder(flinkPod.getPod())
- .withApiVersion(API_VERSION)
- .editOrNewMetadata()
- .withLabels(kubernetesJobManagerParameters.getLabels())
- .withAnnotations(kubernetesJobManagerParameters.getAnnotations())
- .endMetadata()
- .editOrNewSpec()
- .withServiceAccountName(kubernetesJobManagerParameters.getServiceAccount())
- .withImagePullSecrets(kubernetesJobManagerParameters.getImagePullSecrets())
- .withNodeSelector(kubernetesJobManagerParameters.getNodeSelector())
- .withTolerations(
- kubernetesJobManagerParameters.getTolerations().stream()
- .map(
- e ->
- KubernetesToleration.fromMap(e)
- .getInternalResource())
- .collect(Collectors.toList()))
- .endSpec()
- .build();
+ final PodBuilder basicPodBuilder = new PodBuilder(flinkPod.getPodWithoutMainContainer());
+
+ // Overwrite fields
+ final String serviceAccountName =
+ KubernetesUtils.resolveUserDefinedValue(
+ flinkConfig,
+ KubernetesConfigOptions.JOB_MANAGER_SERVICE_ACCOUNT,
+ kubernetesJobManagerParameters.getServiceAccount(),
+ flinkPod.getPodWithoutMainContainer().getSpec().getServiceAccount(),
+ "service account name");
+ if (flinkPod.getPodWithoutMainContainer().getSpec().getRestartPolicy() != null) {
+ logger.info(
+ "The restart policy of JobManager pod will be overwritten to 'always' "
+ + "since it is controlled by the Kubernetes deployment.");
Review comment:
Since we are using the deployment for the Flink JobManager, the restart policy of pod will be overwritten to always anyway. Refer the K8s documentation[1] for more information.
[1]. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pod-template
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581605815
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/utils/KubernetesUtilsTest.java
##########
@@ -75,6 +81,91 @@ public void testCheckWithFixedPort() {
testCheckAndUpdatePortConfigOption("6123", "16123", "6123");
}
+ @Test
+ public void testLoadPodFromTemplateWithNonExistPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ "/path/of/non-exist.yaml",
+ KubernetesPodTemplateTestUtils.TESTING_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNoMainContainerShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesPodTemplateTestUtils.getPodTemplatePath(),
+ "nonExistMainContainer");
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ assertThat(flinkPod.getPod().getSpec().getContainers().size(), is(2));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckMetaData() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesPodTemplateTestUtils.getPodTemplatePath(),
+ KubernetesPodTemplateTestUtils.TESTING_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getMetadata().getName(), is("pod-template"));
Review comment:
I will add a comment here the `pod-template` is specified in `test-resources/testing-pod-template.yaml`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5426ab123aef03d4710c0fea1237fa014684f372 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918)
* 145f891faab2190d83a5ea35ac86a025605b43bd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8d2fd246e442fe92285103908ea6595c64d1c5c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523)
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
}, {
"hash" : "8fd4af27c4e4a62c2ff4207818aa60decbf41934",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13896",
"triggerID" : "8fd4af27c4e4a62c2ff4207818aa60decbf41934",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* 8fd4af27c4e4a62c2ff4207818aa60decbf41934 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13896)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-781924642
cc @xintongsong I have resolved the conflicts and update the documentation based on hugo. Please help with the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-774608130
Thanks @wangyang0918. I'll take a look asap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* a283f5c426e943b90258a061a263733ce2ba62e4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581591845
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -339,6 +339,11 @@ public void close() {
this.internalClient.close();
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(String file) {
+ return new KubernetesPod(this.internalClient.pods().load(file).get());
Review comment:
If the `file` is an invalid file path or the file format is not right, the fabric8 Kubernetes client will fail with exception. Given that we need to fail-hard when the specified pod template file does not exist, I will remove the existence check in `KubernetesUtils#loadPodFromTemplateFile`.
The `load()#get()` will always return a non-null value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145f891faab2190d83a5ea35ac86a025605b43bd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976)
* 6b93e4c21b1edf4cf9c071d88b78ab4230fbc004 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982)
* 04a3a8996bc66bb2d953d5bee380da1a5eff8eac UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773427614
@wangyang0918, I did not have a chance to run the code, but looking at the changes, it looks really good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r582195988
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -60,14 +60,14 @@ public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
new PodBuilder(flinkPod.getPod())
.withApiVersion(API_VERSION)
.editOrNewMetadata()
- .withLabels(kubernetesJobManagerParameters.getLabels())
- .withAnnotations(kubernetesJobManagerParameters.getAnnotations())
+ .addToLabels(kubernetesJobManagerParameters.getLabels())
+ .addToAnnotations(kubernetesJobManagerParameters.getAnnotations())
.endMetadata()
.editOrNewSpec()
.withServiceAccountName(kubernetesJobManagerParameters.getServiceAccount())
Review comment:
I am not sure whether users will find it very ergonomic to look up which options they must configure via config options and which others they can configure via the template. I could imagine that most K8s user will mostly use the template as it is most natural to do it.
I think we should define how the template, explicit Flink configurations, their default values and system values should interact.
I can imagine that we have 1) some values which Flink will overwrite. 2) Then there are settings where Flink will merge its values with the user provided values. 3) And last but not least there are values which overwrite Flink settings.
For 2) and 3) we should define how the user defined values are resolved given that there can be a template setting, an explicitly configured config option and a default value if it has not been set. Moreover, we should define if settings are merged in which order they are merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d191789d3ed2e8715fba7054bd98e773f0a28fde Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043)
* b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r579628708
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/FlinkPod.java
##########
@@ -48,6 +48,12 @@ public Container getMainContainer() {
return mainContainer;
}
+ public FlinkPod clone() {
+ return new FlinkPod(
+ new PodBuilder(this.getPod()).build(),
+ new ContainerBuilder(this.getMainContainer()).build());
Review comment:
Nice suggestion. I will add a new commit for this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MiLk commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
MiLk commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-772181752
Thank you for your work
Following the discussion on the user ML at the beginning of the week, I tried your PR against my workload.
My goal was to set the ephemeral storage requests and limits, but it seems the output of the `ResourceRequirementsBuilder` takes precedence over the template, so my config is ignored.
However I was able to set a `priorityClassName` without trouble.
As discussed in this PR, I think that while options set using flink options should have higher precedence, they should be merged whenever possible with the template.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765023075
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490)
* b8d2fd246e442fe92285103908ea6595c64d1c5c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d191789d3ed2e8715fba7054bd98e773f0a28fde Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043)
* b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-782592922
@xintongsong Thanks for the review. Address your concerns inline and update the PR.
> It seems the current implementation cannot support setting the same template file for both JM / TM, due to the different main container names. Maybe it's not necessary to have different main container names for JM/TM.
Yes, it does not support sharing the pod template at my first implementation. Because I used to believe only the JobManager needs init container. And this will make JobManager and TaskManager pod templates always different. Inspired by you and after more consideration, I think it makes sense to let them share a same pod template since it is easier to use and harmless. I have unified the main container name and introduced a common config option `kubernetes.pod-template-file` in this update.
> It's not clear to me what kind of compatibility guarantees we provide for the pod templates. Would it be possible that something user defined in the template works well in the current Flink version and is overwritten in a new version? I understand it's hard to guarantee that all templates are compatible across versions, but maybe we can define a set of commonly used functions that is guaranteed (guarded by tests) compatible across versions.
I think it is possible that a pod template works well in current Flink version and does not work in a new Flink version. This only happens when we introduce a new config option and overwrite(not merge) the corresponding Kubernetes field. This should rarely happen unless we have no other choices. Then we will document them and publish in the release note.
We could guarantee to our users that a white list fields(e.g. annotations, labels, envs, init container, sidecar container, volumes) in pod template that will always be respected. And I have enriched the tests in `KubernetesJobManagerFactoryWithPodTemplateTest` and `KubernetesTaskManagerFactoryWithPodTemplateTest` to guide them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584563294
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -431,6 +435,61 @@ public static File getTaskManagerPodTemplateFileInPod() {
Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
}
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
+ @Nullable String valueOfPodTemplate,
+ String fieldDescription) {
+ final String resolvedValue;
+ if (valueOfPodTemplate != null) {
+ // The config option is explicitly set.
+ if (flinkConfig.contains(configOption)) {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ LOG.info(
+ "The {} configured in pod template will be overwritten to '{}' "
+ + "because of explicitly configured options.",
+ fieldDescription,
+ resolvedValue);
+ } else {
+ resolvedValue = valueOfPodTemplate;
+ }
+ } else {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ }
+ return resolvedValue;
+ }
+
+ /**
+ * Get the service account from the input pod first, if not specified, the service account name
+ * will be used.
Review comment:
Hmm. Actually, I do not find some concrete differences between them. Both of them are string value and could simply set to the service account name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04a3a8996bc66bb2d953d5bee380da1a5eff8eac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993)
* d191789d3ed2e8715fba7054bd98e773f0a28fde UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765023075
Thanks @wangyang0918,
This works great, I tested it on my app, and its fine. I have 2 questions:
- With this in place, you now have 2 ways for setting annotations, labels, env, resources and tolerations, etc. You can set them using dedicated decorators or through port template yaml., for example:
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/port: "3000"
prometheus.io/scrape: "true"
labels:
deploy: test
version: alphav1
name: pod-template
spec:
containers:
- name: flink-job-manager
env:
- name: "variable1"
value: "value1"
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "2"
memory: 4Gi
volumeMounts:
- mountPath: /opt/flink/conf
name: flink-config-volume
- mountPath: /mnt/flink/storage
name: pvc-checkpointing
- mountPath: /opt/pvcclaim/test/path1
name: pvc-mount2
subPath: path1
- mountPath: /opt/pvcclaim/test/path2
name: pvc-mount2
subPath: path2
- mountPath: /emptydirclaim
name: edir-mount
- mountPath: /var/local/hp
name: hp-mount
- mountPath: /opt/secret
name: secret-mount
- mountPath: /opt/configmap
name: test-configmap
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: cloudflowflink-token-dbnvh
readOnly: true
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
items:
- key: logback-console.xml
path: logback-console.xml
- key: log4j-console.properties
path: log4j-console.properties
- key: flink-conf.yaml
path: flink-conf.yaml
name: flink-config-testapp
name: flink-config-volume
- name: pvc-checkpointing
persistentVolumeClaim:
claimName: testclaim
- name: pvc-mount2
persistentVolumeClaim:
claimName: testclaim1
- emptyDir: {}
name: edir-mount
- hostPath:
path: /var/local/hp
type: DirectoryOrCreate
name: hp-mount
- name: secret-mount
secret:
defaultMode: 420
items:
- key: token
path: value
secretName: test-secret2
- configMap:
defaultMode: 420
items:
- key: key1
path: "1"
name: test-configmap
name: test-configmap
- name: cloudflowflink-token-dbnvh
secret:
defaultMode: 420
secretName: cloudflowflink-token-dbnvh
This template is setting a lot of parameters, that can be used to set a lot of things. Does it make sense to use template as a main configuration approach and remove other options.
- Both this and owner manager PRs are still not merged. - do you have a time frame for merging/releasing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-772197835
@MiLk Thanks for your testing. Could you please share how do you set the request and limit for ephemeral storage, like the following?
```
volumes:
- name: flink-artifact
emptyDir:
sizeLimit: "1Gi"
- name: flink-logs
emptyDir:
sizeLimit: "1Gi"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885)
* a283f5c426e943b90258a061a263733ce2ba62e4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MiLk commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
MiLk commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773664135
I could confirm that I could set both priority class and ephemeral storage requests and limits 🎉
It seems to behave as expected, thank you for the hard work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673)
* 2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r582506793
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManagerDriver.java
##########
@@ -100,6 +104,17 @@ protected void initializeInternal() throws Exception {
kubeClientOpt =
Optional.of(kubeClientFactory.fromConfiguration(flinkConfig, getIoExecutor()));
podsWatchOpt = watchTaskManagerPods();
+ taskManagerPodTemplate =
+ flinkConfig
+ .getOptional(KubernetesConfigOptions.TASK_MANAGER_POD_TEMPLATE)
+ .map(
+ ignore ->
+ KubernetesUtils.loadPodFromTemplateFile(
+ kubeClientOpt.get(),
+ KubernetesUtils.getTaskManagerPodTemplateFileInPod(
+ flinkConfig),
+ Constants.MAIN_CONTAINER_NAME))
+ .orElse(new FlinkPod.Builder().build());
Review comment:
Make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765023075
Thanks @wangyang0918,
This works great, I tested it on my app, and its fine. I have 2 questions:
- With this in place, you now have 2 ways for setting annotations, labels, env, resources and tolerations, etc. You can set them using dedicated decorators or through port template yaml., for example:
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/port: "3000"
prometheus.io/scrape: "true"
labels:
deploy: test
version: alphav1
name: pod-template
spec:
containers:
- name: flink-job-manager
env:
- name: "variable1"
value: "value1"
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "2"
memory: 4Gi
volumeMounts:
- mountPath: /opt/flink/conf
name: flink-config-volume
- mountPath: /mnt/flink/storage
name: pvc-checkpointing
- mountPath: /opt/pvcclaim/test/path1
name: pvc-mount2
subPath: path1
- mountPath: /opt/pvcclaim/test/path2
name: pvc-mount2
subPath: path2
- mountPath: /emptydirclaim
name: edir-mount
- mountPath: /var/local/hp
name: hp-mount
- mountPath: /opt/secret
name: secret-mount
- mountPath: /opt/configmap
name: test-configmap
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: cloudflowflink-token-dbnvh
readOnly: true
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
items:
- key: logback-console.xml
path: logback-console.xml
- key: log4j-console.properties
path: log4j-console.properties
- key: flink-conf.yaml
path: flink-conf.yaml
name: flink-config-testapp
name: flink-config-volume
- name: pvc-checkpointing
persistentVolumeClaim:
claimName: testclaim
- name: pvc-mount2
persistentVolumeClaim:
claimName: testclaim1
- emptyDir: {}
name: edir-mount
- hostPath:
path: /var/local/hp
type: DirectoryOrCreate
name: hp-mount
- name: secret-mount
secret:
defaultMode: 420
items:
- key: token
path: value
secretName: test-secret2
- configMap:
defaultMode: 420
items:
- key: key1
path: "1"
name: test-configmap
name: test-configmap
- name: cloudflowflink-token-dbnvh
secret:
defaultMode: 420
secretName: cloudflowflink-token-dbnvh
Template allows setting a lot of parameters and is a very simple and convenient mechanism. Does it make sense to use template as a main configuration approach and remove other options. What I am afraid of is people setting them in 2 different places and then chasing where the setting came from or overwriting settings
- Both this and owner manager PRs are still not merged. - do you have a time frame for merging/releasing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r582060108
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -60,14 +60,14 @@ public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
new PodBuilder(flinkPod.getPod())
.withApiVersion(API_VERSION)
.editOrNewMetadata()
- .withLabels(kubernetesJobManagerParameters.getLabels())
- .withAnnotations(kubernetesJobManagerParameters.getAnnotations())
+ .addToLabels(kubernetesJobManagerParameters.getLabels())
+ .addToAnnotations(kubernetesJobManagerParameters.getAnnotations())
.endMetadata()
.editOrNewSpec()
.withServiceAccountName(kubernetesJobManagerParameters.getServiceAccount())
Review comment:
When I design the pod template at the very begging, I have a simple rule that pod template is used to configure the advanced features(e.g. init container, sidecar container, volumes, etc) which we have not supported via config options. So if a Kubernetes feature has a corresponding config option, the users should use it. If it is not explicitly configured, the default value will take effect. This will force the users to make the pod template as simple as possible.
This is just my idea. Your suggestion about the precedence also makes sense. I will have a try on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581608052
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
Review comment:
The only reason why I introduce this abstract class is to avoid duplicated tests for `KubernetesJobManagerFactoryWithPodTemplateTest` and `KubernetesTaskManagerFactoryWithPodTemplateTest`. It is not about setting up the main part.
The implementation classes need to `setResultPod` and then the result pod will be verified in the base class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584128985
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,240 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files. This allows to support advanced features
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains the pod definition. It will be used to initialize the JobManager and TaskManager.
+The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template will be overwritten by Flink.
+The mechanism for resolving effective field values can be categorized as follows:
+* **Defined by Flink:** User cannot configure it.
+* **Defined by the user:** User can freely specify this value. Flink framework won't set any additional values and the effective value derives from the config option and the template.
+
+ Precedence order: First an explicit config option value is taken, then the value in pod template and at last the default value of a config option if nothing is specified.
+* **Merged with Flink:** Flink will merge values for a setting with a user defined value (see precedence order for "Defined by the user"). Flink values have precedence in case of same name fields.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
Review comment:
Maybe the owner reference of pod should be "Defined by Flink". Actually the users could not set the owner reference of JobManager and TaskManager pods. Because the JobManager pod owner reference will be always overwritten to the JobManager deployment. And so is the TaskManager pod.
Users could only control the owner reference of JobManager deployment, which means when the whole Flink cluster will be deleted.
I will also update all other the description of "Defined by the user".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787088549
cc @tillrohrmann comments addressed. Please have a look at your convenience.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765023075
Thanks @wangyang0918,
This works great, I tested it on my app, and its fine. I have 2 questions:
- With this in place, you now have 2 ways for setting annotations, labels, env, resources and tolerations, etc. You can set them using dedicated decorators or through port template yaml., for example:
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/port: "3000"
prometheus.io/scrape: "true"
labels:
deploy: test
version: alphav1
name: pod-template
spec:
containers:
- name: flink-job-manager
env:
- name: "variable1"
value: "value1"
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "2"
memory: 4Gi
volumeMounts:
- mountPath: /opt/flink/conf
name: flink-config-volume
- mountPath: /mnt/flink/storage
name: pvc-checkpointing
- mountPath: /opt/pvcclaim/test/path1
name: pvc-mount2
subPath: path1
- mountPath: /opt/pvcclaim/test/path2
name: pvc-mount2
subPath: path2
- mountPath: /emptydirclaim
name: edir-mount
- mountPath: /var/local/hp
name: hp-mount
- mountPath: /opt/secret
name: secret-mount
- mountPath: /opt/configmap
name: test-configmap
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: cloudflowflink-token-dbnvh
readOnly: true
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
items:
- key: logback-console.xml
path: logback-console.xml
- key: log4j-console.properties
path: log4j-console.properties
- key: flink-conf.yaml
path: flink-conf.yaml
name: flink-config-testapp
name: flink-config-volume
- name: pvc-checkpointing
persistentVolumeClaim:
claimName: testclaim
- name: pvc-mount2
persistentVolumeClaim:
claimName: testclaim1
- emptyDir: {}
name: edir-mount
- hostPath:
path: /var/local/hp
type: DirectoryOrCreate
name: hp-mount
- name: secret-mount
secret:
defaultMode: 420
items:
- key: token
path: value
secretName: test-secret2
- configMap:
defaultMode: 420
items:
- key: key1
path: "1"
name: test-configmap
name: test-configmap
- name: cloudflowflink-token-dbnvh
secret:
defaultMode: 420
secretName: cloudflowflink-token-dbnvh
This template allows setting a lot of parameters and is a very simple and convenient mechanism. Does it make sense to use template as a main configuration approach and remove other options. What I am afraid of is people setting them in 2 different places and then chasing where the setting came from or overwriting settings
- Both this and owner manager PRs are still not merged. - do you have a time frame for merging/releasing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d191789d3ed2e8715fba7054bd98e773f0a28fde Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 removed a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 removed a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787104579
cc @tillrohrmann Comments addressed. Please have a look at your convenience.
I have also changed a bit about the service account because of following reasons.
* The Kubernetes provides two ways to set the service account(`withServiceAccount` and `withServiceAccountName`)
* When checking whether config options are explicitly set, the fallback config option should also be checked. So I am introducing the `withFallbackKeys`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r583726233
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -49,64 +50,104 @@
public class InitJobManagerDecorator extends AbstractKubernetesStepDecorator {
private final KubernetesJobManagerParameters kubernetesJobManagerParameters;
+ private final Configuration flinkConfig;
public InitJobManagerDecorator(KubernetesJobManagerParameters kubernetesJobManagerParameters) {
this.kubernetesJobManagerParameters = checkNotNull(kubernetesJobManagerParameters);
+ this.flinkConfig = checkNotNull(kubernetesJobManagerParameters.getFlinkConfiguration());
}
@Override
public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
- final Pod basicPod =
- new PodBuilder(flinkPod.getPod())
- .withApiVersion(API_VERSION)
- .editOrNewMetadata()
- .withLabels(kubernetesJobManagerParameters.getLabels())
- .withAnnotations(kubernetesJobManagerParameters.getAnnotations())
- .endMetadata()
- .editOrNewSpec()
- .withServiceAccountName(kubernetesJobManagerParameters.getServiceAccount())
- .withImagePullSecrets(kubernetesJobManagerParameters.getImagePullSecrets())
- .withNodeSelector(kubernetesJobManagerParameters.getNodeSelector())
- .withTolerations(
- kubernetesJobManagerParameters.getTolerations().stream()
- .map(
- e ->
- KubernetesToleration.fromMap(e)
- .getInternalResource())
- .collect(Collectors.toList()))
- .endSpec()
- .build();
+ final PodBuilder basicPodBuilder = new PodBuilder(flinkPod.getPodWithoutMainContainer());
+
+ // Overwrite fields
+ final String serviceAccountName =
+ KubernetesUtils.resolveUserDefinedValue(
+ flinkConfig,
+ KubernetesConfigOptions.JOB_MANAGER_SERVICE_ACCOUNT,
+ kubernetesJobManagerParameters.getServiceAccount(),
+ flinkPod.getPodWithoutMainContainer().getSpec().getServiceAccount(),
+ "service account name");
+ if (flinkPod.getPodWithoutMainContainer().getSpec().getRestartPolicy() != null) {
+ logger.info(
+ "The restart policy of JobManager pod will be overwritten to 'always' "
+ + "since it is controlled by the Kubernetes deployment.");
Review comment:
Is `always` the default or why don't we set it explicity via `withRestartPolicy("Always")`? It just a bit confusing that I don't see the builder call further down.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitTaskManagerDecorator.java
##########
@@ -41,64 +42,108 @@
public class InitTaskManagerDecorator extends AbstractKubernetesStepDecorator {
private final KubernetesTaskManagerParameters kubernetesTaskManagerParameters;
+ private final Configuration flinkConfig;
public InitTaskManagerDecorator(
KubernetesTaskManagerParameters kubernetesTaskManagerParameters) {
this.kubernetesTaskManagerParameters = checkNotNull(kubernetesTaskManagerParameters);
+ this.flinkConfig = checkNotNull(kubernetesTaskManagerParameters.getFlinkConfiguration());
}
@Override
public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
- final Pod basicPod =
- new PodBuilder(flinkPod.getPod())
- .withApiVersion(Constants.API_VERSION)
- .editOrNewMetadata()
- .withName(kubernetesTaskManagerParameters.getPodName())
- .withLabels(kubernetesTaskManagerParameters.getLabels())
- .withAnnotations(kubernetesTaskManagerParameters.getAnnotations())
- .endMetadata()
- .editOrNewSpec()
- .withServiceAccountName(kubernetesTaskManagerParameters.getServiceAccount())
- .withRestartPolicy(Constants.RESTART_POLICY_OF_NEVER)
- .withImagePullSecrets(kubernetesTaskManagerParameters.getImagePullSecrets())
- .withNodeSelector(kubernetesTaskManagerParameters.getNodeSelector())
- .withTolerations(
- kubernetesTaskManagerParameters.getTolerations().stream()
- .map(
- e ->
- KubernetesToleration.fromMap(e)
- .getInternalResource())
- .collect(Collectors.toList()))
- .endSpec()
- .build();
+ final PodBuilder basicPodBuilder = new PodBuilder(flinkPod.getPodWithoutMainContainer());
+
+ // Overwrite fields
+ final String serviceAccountName =
+ KubernetesUtils.resolveUserDefinedValue(
+ flinkConfig,
+ KubernetesConfigOptions.JOB_MANAGER_SERVICE_ACCOUNT,
Review comment:
```suggestion
KubernetesConfigOptions.TASK_MANAGER_SERVICE_ACCOUNT,
```
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +405,74 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, File podTemplateFile, String mainContainerName) {
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(podTemplateFile);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.info(
+ "Could not find main container {} in pod template, using empty one to initialize.",
+ mainContainerName);
+ mainContainer = new ContainerBuilder().build();
+ }
+
+ pod.getInternalResource().getSpec().setContainers(otherContainers);
+ return new FlinkPod(pod.getInternalResource(), mainContainer);
+ }
+
+ public static File getTaskManagerPodTemplateFileInPod() {
+ return new File(
+ Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
+ }
+
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
Review comment:
Can we extract this value from `flinkConfig.get(configOption)`? This might make the method call a bit easier.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,240 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files. This allows to support advanced features
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains the pod definition. It will be used to initialize the JobManager and TaskManager.
+The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template will be overwritten by Flink.
+The mechanism for resolving effective field values can be categorized as follows:
+* **Defined by Flink:** User cannot configure it.
+* **Defined by the user:** User can freely specify this value. Flink framework won't set any additional values and the effective value derives from the config option and the template.
+
+ Precedence order: First an explicit config option value is taken, then the value in pod template and at last the default value of a config option if nothing is specified.
+* **Merged with Flink:** Flink will merge values for a setting with a user defined value (see precedence order for "Defined by the user"). Flink values have precedence in case of same name fields.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
Review comment:
Maybe say that the value is resolved with respect to the defined precedence order for user defined values. Now it sounds as if the configuration option is the only way to define the value. This actually applies to all "Defined by the user" values.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,240 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files. This allows to support advanced features
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains the pod definition. It will be used to initialize the JobManager and TaskManager.
+The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template will be overwritten by Flink.
+The mechanism for resolving effective field values can be categorized as follows:
+* **Defined by Flink:** User cannot configure it.
+* **Defined by the user:** User can freely specify this value. Flink framework won't set any additional values and the effective value derives from the config option and the template.
+
+ Precedence order: First an explicit config option value is taken, then the value in pod template and at last the default value of a config option if nothing is specified.
+* **Merged with Flink:** Flink will merge values for a setting with a user defined value (see precedence order for "Defined by the user"). Flink values have precedence in case of same name fields.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Merged with Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add some internal labels to the user defined values.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Pod Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>"always" for JobManager pod and "never" for TaskManager pod.
+ <br>
+ The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a></td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>Merged with Flink</td>
+ <td></td>
+ <td>Flink will add some internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Main Container Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Merged with Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add some internal environment variables to the user defined values.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The container image will be overwritten by Flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>imagePullPolicy</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-policy">kubernetes.container.image.pull-policy</a></td>
+ <td>The container image pull policy will be overwritten by Flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The container name will be overwritten by Flink with "flink-main-container".</td>
+ </tr>
+ <tr>
+ <td>resources</td>
+ <td>Defined by the user</td>
+ <td>Memory: <br>
+ <a href="{{< ref "docs/deployment/config" >}}#jobmanager-memory-process-size">jobmanager.memory.process.size</a>
+ <a href="{{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size">taskmanager.memory.process.size</a>
+ <br>
+ CPU: <br>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-cpu">kubernetes.jobmanager.cpu</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-cpu">kubernetes.taskmanager.cpu</a></td>
+ <td>The memory and cpu resources(including requests and limits) will be overwritten by Flink. All other resources(e.g. ephemeral-storage) will be retained.</td>
Review comment:
Did you mean this?
```suggestion
<td>The memory and cpu resources(including requests and limits) will be overwritten by Flink configuration options. All other resources(e.g. ephemeral-storage) will be retained.</td>
```
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/decorators/PodTemplateMountDecoratorTest.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.decorators;
+
+import org.apache.flink.core.testutils.FlinkMatchers;
+import org.apache.flink.kubernetes.KubernetesTestUtils;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptions;
+import org.apache.flink.kubernetes.kubeclient.FlinkPod;
+import org.apache.flink.kubernetes.kubeclient.KubernetesJobManagerTestBase;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.ConfigMap;
+import io.fabric8.kubernetes.api.model.HasMetadata;
+import io.fabric8.kubernetes.api.model.KeyToPath;
+import io.fabric8.kubernetes.api.model.KeyToPathBuilder;
+import io.fabric8.kubernetes.api.model.Volume;
+import io.fabric8.kubernetes.api.model.VolumeBuilder;
+import io.fabric8.kubernetes.api.model.VolumeMount;
+import io.fabric8.kubernetes.api.model.VolumeMountBuilder;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.kubernetes.utils.Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
+
+/** General tests for the {@link PodTemplateMountDecorator}. */
+public class PodTemplateMountDecoratorTest extends KubernetesJobManagerTestBase {
+
+ private static final String POD_TEMPLATE_FILE_NAME = "testing-pod-template.yaml";
+ private static final String POD_TEMPLATE_DATA = "taskmanager pod template data";
+
+ private PodTemplateMountDecorator podTemplateMountDecorator;
+
+ @Override
+ protected void setupFlinkConfig() {
+ super.setupFlinkConfig();
+
+ this.flinkConfig.set(
+ KubernetesConfigOptions.TASK_MANAGER_POD_TEMPLATE,
+ new File(flinkConfDir, POD_TEMPLATE_FILE_NAME).getAbsolutePath());
+ }
+
+ @Override
+ protected void onSetup() throws Exception {
+ super.onSetup();
+
+ this.podTemplateMountDecorator =
+ new PodTemplateMountDecorator(kubernetesJobManagerParameters);
+ }
+
+ @Test
+ public void testBuildAccompanyingKubernetesResourcesAddsPodTemplateAsConfigMap()
+ throws IOException {
+ KubernetesTestUtils.createTemporyFile(
+ POD_TEMPLATE_DATA, flinkConfDir, POD_TEMPLATE_FILE_NAME);
+
+ final List<HasMetadata> additionalResources =
+ podTemplateMountDecorator.buildAccompanyingKubernetesResources();
+ assertThat(additionalResources.size(), is(1));
+
+ final ConfigMap resultConfigMap = (ConfigMap) additionalResources.get(0);
+
+ Map<String, String> resultDatas = resultConfigMap.getData();
Review comment:
```suggestion
Map<String, String> resultData = resultConfigMap.getData();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
* 2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MiLk commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
MiLk commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584584234
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -431,6 +435,61 @@ public static File getTaskManagerPodTemplateFileInPod() {
Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
}
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
+ @Nullable String valueOfPodTemplate,
+ String fieldDescription) {
+ final String resolvedValue;
+ if (valueOfPodTemplate != null) {
+ // The config option is explicitly set.
+ if (flinkConfig.contains(configOption)) {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ LOG.info(
+ "The {} configured in pod template will be overwritten to '{}' "
+ + "because of explicitly configured options.",
+ fieldDescription,
+ resolvedValue);
+ } else {
+ resolvedValue = valueOfPodTemplate;
+ }
+ } else {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ }
+ return resolvedValue;
+ }
+
+ /**
+ * Get the service account from the input pod first, if not specified, the service account name
+ * will be used.
Review comment:
`serviceAccount` has been deprecated a while ago and replace by `serviceAccountName`.
https://github.com/kubernetes/kubernetes/commit/7a3891e5f8819456b355750c0603e48da35b895b
https://github.com/kubernetes/kubernetes/pull/10081
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -431,6 +435,61 @@ public static File getTaskManagerPodTemplateFileInPod() {
Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
}
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
+ @Nullable String valueOfPodTemplate,
+ String fieldDescription) {
+ final String resolvedValue;
+ if (valueOfPodTemplate != null) {
+ // The config option is explicitly set.
+ if (flinkConfig.contains(configOption)) {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ LOG.info(
+ "The {} configured in pod template will be overwritten to '{}' "
+ + "because of explicitly configured options.",
+ fieldDescription,
+ resolvedValue);
+ } else {
+ resolvedValue = valueOfPodTemplate;
+ }
+ } else {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ }
+ return resolvedValue;
+ }
+
+ /**
+ * Get the service account from the input pod first, if not specified, the service account name
+ * will be used.
Review comment:
`serviceAccount` has been deprecated a while ago and replaced by `serviceAccountName`.
https://github.com/kubernetes/kubernetes/commit/7a3891e5f8819456b355750c0603e48da35b895b
https://github.com/kubernetes/kubernetes/pull/10081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-785978493
@tillrohrmann Thanks for your comments and suggestions. I have updated this PR. The main changes are list as followings.
* Add a separate decorator for `PodTemplateMountDecorator` which is only used for `KubernetesJobManagerFactory`. And then we could still keep the file existence check.
* Introduce new mechanism for resolving effective field values. "Defined by Flink", "Defined by the user" and "Merged with Flink". The documentation is updated accordingly.
For your posted two problems, the first one is a bug and has been fixed. For the second one, I think it is an expected behavior. We have deferred the sanity check to the Fabric8 Kubernetes client, which will call the Kubernetes APIServer for the verification. I admin that the exception is not friendly, but you still could find the root cause `spec.template.spec.containers[0].image: Required value`. I guess you do not set the image for the non-main container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
* 2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581600918
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
+
+ private static final String ENTRY_POINT_CLASS =
+ KubernetesSessionClusterEntrypoint.class.getCanonicalName();
+ private static final int RESOURCE_MEMORY = 1456;
+
+ /**
+ * The result pod variables should be set via {@link #setResultPod()} by implementation class.
+ */
+ protected ObjectMeta resultPodMeta;
+
+ protected PodSpec resultPodSpec;
Review comment:
I will rename the `setResultPod` to `getResultPod` and make these variables private.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584622922
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -431,6 +435,61 @@ public static File getTaskManagerPodTemplateFileInPod() {
Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
}
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
+ @Nullable String valueOfPodTemplate,
+ String fieldDescription) {
+ final String resolvedValue;
+ if (valueOfPodTemplate != null) {
+ // The config option is explicitly set.
+ if (flinkConfig.contains(configOption)) {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ LOG.info(
+ "The {} configured in pod template will be overwritten to '{}' "
+ + "because of explicitly configured options.",
+ fieldDescription,
+ resolvedValue);
+ } else {
+ resolvedValue = valueOfPodTemplate;
+ }
+ } else {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ }
+ return resolvedValue;
+ }
+
+ /**
+ * Get the service account from the input pod first, if not specified, the service account name
+ * will be used.
Review comment:
Thanks for sharing this information. Currently, in Flink, we have set the `serviceAccount` and `serviceAccountName` both.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r582080530
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
Review comment:
Yes. This is the current design and implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581608052
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
Review comment:
The only reason why I introduce this abstract class is to avoid duplicated tests for `KubernetesJobManagerFactoryWithPodTemplateTest` and `KubernetesTaskManagerFactoryWithPodTemplateTest`. It is not about setting up the main part.
Maybe I could rename the `setResultPod` to `getResult` in the base class and force the implementation to construct the result pod.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r580876622
##########
File path: docs/content/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name "flink-main-container".
Review comment:
```suggestion
JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
```
##########
File path: docs/content/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name "flink-main-container".
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+##### Pod Metadata
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+##### Pod Spec
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>"always" for JobManager pod and "never" for TaskManager pod.
+ <br>
+ The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a></td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>Overwritten by Flink</td>
+ <td></td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+##### Main Container Spec
Review comment:
```suggestion
**Main Container Spec**
```
##########
File path: docs/content/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name "flink-main-container".
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+##### Pod Metadata
Review comment:
```suggestion
**Pod Metadata**
```
##########
File path: docs/content/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name "flink-main-container".
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+##### Pod Metadata
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+##### Pod Spec
Review comment:
```suggestion
**Pod Spec**
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8d2fd246e442fe92285103908ea6595c64d1c5c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523)
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623)
* 021dd8790e62582df174355db6ad1ad978230c65 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631)
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 removed a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 removed a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787088549
cc @tillrohrmann comments addressed. Please have a look at your convenience.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581591845
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -339,6 +339,11 @@ public void close() {
this.internalClient.close();
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(String file) {
+ return new KubernetesPod(this.internalClient.pods().load(file).get());
Review comment:
If the `file` is an invalid file path or the file format is not right, the fabric8 Kubernetes client will fail with exception. Given that this is uncertain assumption, I will move the existence check from `KubernetesUtils#loadPodFromTemplateFile` to here.
The `load()#get()` will always return a non-null value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584124388
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +405,74 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, File podTemplateFile, String mainContainerName) {
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(podTemplateFile);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.info(
+ "Could not find main container {} in pod template, using empty one to initialize.",
+ mainContainerName);
+ mainContainer = new ContainerBuilder().build();
+ }
+
+ pod.getInternalResource().getSpec().setContainers(otherContainers);
+ return new FlinkPod(pod.getInternalResource(), mainContainer);
+ }
+
+ public static File getTaskManagerPodTemplateFileInPod() {
+ return new File(
+ Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
+ }
+
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
Review comment:
The reason I do not extract this value from `flinkConfig.get(configOption)` because I want to extract them from `KubernetesParameters`. Then we could reuse the pre condition check(e.g. empty string, length, maybe more in the future) and fallback logic(e.g. `getServiceAccount`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c06cad57cc46b5e84184ae916138cdf5ae027d4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680)
* eeea5dc67837785f54f7e18151095f2693a7d391 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #14629:
URL: https://github.com/apache/flink/pull/14629
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623)
* 021dd8790e62582df174355db6ad1ad978230c65 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04a3a8996bc66bb2d953d5bee380da1a5eff8eac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r582076000
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -98,7 +103,7 @@ private Container decorateMainContainer(Container container) {
.withImagePullPolicy(kubernetesJobManagerParameters.getImagePullPolicy().name())
.withResources(requirements)
.withPorts(getContainerPorts())
Review comment:
Just like what we have described in the documentation, some fields will always be respected. Flink provides the following list of fields that are planned to be long-term supported: `annotations, labels, imagePullSecrets, nodeSelector, tolerations, env, init container, sidecar container, volumes`. The fields that will be overwritten are listed in the tables.
Adding a sanity check for the pod template in one method does not make sense to me. Because it will also be a burden to maintain especially when we change the internal overwrite logic or introduce new config options. I will try to figure out whether it is feasible to log whenever we overwrite a pod template configuration in decorators.
Currently, we already log the result pod meta and spec in the `Fabric8FlinkKubeClient` when creating JobManager and TaskManager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5426ab123aef03d4710c0fea1237fa014684f372 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581594031
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, String path, String mainContainerName) {
+ if (!new File(path).exists()) {
+ return new FlinkPod.Builder().build();
Review comment:
Fail hard makes sense. I will move the existence check to `Fabric8KubernetesClient#loadPodFromTemplateFile`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581240978
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/FlinkPod.java
##########
@@ -48,6 +48,13 @@ public Container getMainContainer() {
return mainContainer;
}
+ @Override
+ public FlinkPod clone() {
Review comment:
Do we really want to override `Object.clone` here? Java's clone contract is super hard to adhere to. If it is only about creating a method which creates a copy of `FlinkPod` I would suggest to add a `copy` method.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##########
@@ -367,6 +369,54 @@
code("FlinkKubeClient#checkAndUpdateConfigMap"))
.build());
+ public static final ConfigOption<String> JOB_MANAGER_POD_TEMPLATE =
+ key("kubernetes.pod-template-file.jobmanager")
+ .stringType()
+ .noDefaultValue()
+ .withFallbackKeys(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .withDescription(
+ "Specify a local file that contains the jobmanager pod template definition. "
+ + "It will be used to initialize the jobmanager pod. "
+ + "The main container should be defined with name '"
+ + Constants.MAIN_CONTAINER_NAME
+ + "'. If not explicitly configured, config option '"
+ + KUBERNETES_POD_TEMPLATE_FILE_KEY
+ + "' will be used.");
+
+ public static final ConfigOption<String> TASK_MANAGER_POD_TEMPLATE =
+ key("kubernetes.pod-template-file.taskmanager")
Review comment:
Same here.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -339,6 +339,11 @@ public void close() {
this.internalClient.close();
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(String file) {
+ return new KubernetesPod(this.internalClient.pods().load(file).get());
Review comment:
What happens if `file` is an invalid file path?
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesPodTemplateTestUtils.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ContainerBuilder;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import io.fabric8.kubernetes.api.model.Volume;
+import io.fabric8.kubernetes.api.model.VolumeBuilder;
+import io.fabric8.kubernetes.api.model.VolumeMount;
+import io.fabric8.kubernetes.api.model.VolumeMountBuilder;
+
+import java.net.URL;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.junit.Assert.assertThat;
+
+/** Utilities for the Kubernetes pod template tests. */
+public class KubernetesPodTemplateTestUtils {
+
+ public static final String TESTING_MAIN_CONTAINER_NAME = "testing-main-container";
+ public static final String TESTING_SIDE_CAR_CONTAINER_NAME = "sidecar-log-collector";
+
+ private static final String TESTING_TEMPLATE_FILE_NAME = "testing-pod-template.yaml";
+
+ public static String getPodTemplatePath() {
+ final URL podTemplateUrl =
+ KubernetesPodTemplateTestUtils.class
+ .getClassLoader()
+ .getResource(TESTING_TEMPLATE_FILE_NAME);
+ assertThat(podTemplateUrl, not(nullValue()));
+ return podTemplateUrl.getPath();
+ }
+
+ public static Container getInitContainerInPodTemplate() {
Review comment:
maybe `createInitContainerForPodTemplate` or `createInitContainer`.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/decorators/FlinkConfMountDecoratorTest.java
##########
@@ -243,4 +220,63 @@ public void testDecoratedFlinkContainer() {
assertEquals(Constants.FLINK_CONF_VOLUME, volumeMount.getName());
assertEquals(FLINK_CONF_DIR_IN_POD, volumeMount.getMountPath());
}
+
+ @Test
+ public void testConfigMapWithTaskManagerPodTemplate() throws IOException {
Review comment:
Maybe call it `testBuildAccompanyingKubernetesResourcesAddsPodTemplateAsConfigMap`
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
+
+ private static final String ENTRY_POINT_CLASS =
+ KubernetesSessionClusterEntrypoint.class.getCanonicalName();
+ private static final int RESOURCE_MEMORY = 1456;
+
+ /**
+ * The result pod variables should be set via {@link #setResultPod()} by implementation class.
+ */
+ protected ObjectMeta resultPodMeta;
+
+ protected PodSpec resultPodSpec;
+
+ @Override
+ protected void setupFlinkConfig() {
+ super.setupFlinkConfig();
+ flinkConfig.set(KubernetesConfigOptionsInternal.ENTRY_POINT_CLASS, ENTRY_POINT_CLASS);
+
+ // Set fixed ports
+ flinkConfig.set(RestOptions.PORT, Constants.REST_PORT);
+ flinkConfig.set(BlobServerOptions.PORT, Integer.toString(Constants.BLOB_SERVER_PORT));
+ flinkConfig.setString(
+ TaskManagerOptions.RPC_PORT, String.valueOf(Constants.TASK_MANAGER_RPC_PORT));
+
+ flinkConfig.set(
+ TaskManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.ofMebiBytes(RESOURCE_MEMORY));
+ }
+
+ @Override
+ public final void onSetup() throws Exception {
+ setResultPod();
+ }
+
+ protected abstract void setResultPod() throws Exception;
+
+ @Test
+ public void testInitContainerFromPodTemplate() {
+ assertThat(resultPodSpec.getInitContainers().size(), is(1));
+ assertThat(
+ resultPodSpec.getInitContainers().get(0),
+ is(KubernetesPodTemplateTestUtils.getInitContainerInPodTemplate()));
+ }
+
+ @Test
+ public void testSideCarContainerFromPodTemplate() {
+ final Container sideCarContainer =
+ KubernetesPodTemplateTestUtils.getContainerWithName(
+ resultPodSpec,
+ KubernetesPodTemplateTestUtils.TESTING_SIDE_CAR_CONTAINER_NAME);
+ assertThat(
+ sideCarContainer,
+ is(KubernetesPodTemplateTestUtils.getSideCarContainerInPodTemplate()));
+ }
+
+ @Test
+ public void testVolumesFromPodTemplate() {
+ assertThat(
+ resultPodSpec.getVolumes(),
+ hasItems(KubernetesPodTemplateTestUtils.getVolumesInPodTemplate()));
+ }
+
+ @Test
+ public void testMainContainerVolumeMountsFromPodTemplate() {
+ final Container mainContainer =
+ KubernetesPodTemplateTestUtils.getContainerWithName(
+ resultPodSpec, Constants.MAIN_CONTAINER_NAME);
+ assertThat(
+ mainContainer.getVolumeMounts(),
+ hasItems(KubernetesPodTemplateTestUtils.getVolumeMountInPodTemplate()));
+ }
Review comment:
Looking at the assertions in these tests it is not clear to me why the expected values are what they are. Only when looking at the subclasses of this class one sees that `KubernetesPodTemplateTestUtils.getPodTemplatePath()` is used. This contract is not clear and can easily break. It is much easier to reason about tests if one sees how the test subject is set up and the assertions in one method.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/Constants.java
##########
@@ -97,4 +97,7 @@
public static final String CHECKPOINT_COUNTER_KEY = "counter";
public static final String CHECKPOINT_ID_KEY_PREFIX = "checkpointID-";
public static final String COMPLETED_CHECKPOINT_FILE_SUFFIX = "completedCheckpoint";
+
+ public static final String TASK_MANAGER_POD_TEMPLATE_FILE_NAME =
+ "taskmanager-pod-template.yaml";
Review comment:
Maybe add a comment that this is the path to the mounted task manager pod template in the JobManager pod if there was any specified.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, String path, String mainContainerName) {
+ if (!new File(path).exists()) {
+ return new FlinkPod.Builder().build();
Review comment:
Maybe add a log statement that we default to the standard `FlinkPod` because `path` could not be read.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##########
@@ -367,6 +369,54 @@
code("FlinkKubeClient#checkAndUpdateConfigMap"))
.build());
+ public static final ConfigOption<String> JOB_MANAGER_POD_TEMPLATE =
+ key("kubernetes.pod-template-file.jobmanager")
Review comment:
nit: you could use `KUBERNETES_POD_TEMPLATE_FILE_KEY + ".jobmanager"`
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, String path, String mainContainerName) {
+ if (!new File(path).exists()) {
+ return new FlinkPod.Builder().build();
+ }
+
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(path);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.warn(
Review comment:
I would make this `info` because it is a valid use case that the user does not want to modify the main container, right? Or is it required that the user specifies a main container?
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, String path, String mainContainerName) {
Review comment:
nit changing `path` to `File` type could let us avoid converting between `File -> String -> File` when using the `getTaskManagerPodTemplatePathInPod` and it gives us a bit more of type safety.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesPodTemplateTestUtils.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ContainerBuilder;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import io.fabric8.kubernetes.api.model.Volume;
+import io.fabric8.kubernetes.api.model.VolumeBuilder;
+import io.fabric8.kubernetes.api.model.VolumeMount;
+import io.fabric8.kubernetes.api.model.VolumeMountBuilder;
+
+import java.net.URL;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.junit.Assert.assertThat;
+
+/** Utilities for the Kubernetes pod template tests. */
+public class KubernetesPodTemplateTestUtils {
+
+ public static final String TESTING_MAIN_CONTAINER_NAME = "testing-main-container";
+ public static final String TESTING_SIDE_CAR_CONTAINER_NAME = "sidecar-log-collector";
+
+ private static final String TESTING_TEMPLATE_FILE_NAME = "testing-pod-template.yaml";
+
+ public static String getPodTemplatePath() {
+ final URL podTemplateUrl =
+ KubernetesPodTemplateTestUtils.class
+ .getClassLoader()
+ .getResource(TESTING_TEMPLATE_FILE_NAME);
+ assertThat(podTemplateUrl, not(nullValue()));
+ return podTemplateUrl.getPath();
+ }
+
+ public static Container getInitContainerInPodTemplate() {
+ return new ContainerBuilder()
+ .withName("artifacts-fetcher")
+ .withImage("testing-init-image")
+ .withCommand(
+ "wget",
+ "https://path/of/StateMachineExample.jar",
+ "-O",
+ "/flink-artifact/myjob.jar")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-artifact")
+ .withMountPath("/flink-artifact")
+ .build())
+ .build();
+ }
+
+ public static Container getSideCarContainerInPodTemplate() {
Review comment:
`createSideCarContainerForPodTemplate` or `createSideCarContainer`.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
Review comment:
In general I would be cautious with using inheritance for test classes. This makes it a lot harder for someone new to this part of the code to understand what is actually happening. In most cases, one can avoid test bases by using composition (e.g. creating a `KubernetesTestingResource` which sets up the main part and can be used by test clases which want to test something for K8s).
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesPodTemplateTestUtils.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ContainerBuilder;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import io.fabric8.kubernetes.api.model.Volume;
+import io.fabric8.kubernetes.api.model.VolumeBuilder;
+import io.fabric8.kubernetes.api.model.VolumeMount;
+import io.fabric8.kubernetes.api.model.VolumeMountBuilder;
+
+import java.net.URL;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.junit.Assert.assertThat;
+
+/** Utilities for the Kubernetes pod template tests. */
+public class KubernetesPodTemplateTestUtils {
+
+ public static final String TESTING_MAIN_CONTAINER_NAME = "testing-main-container";
+ public static final String TESTING_SIDE_CAR_CONTAINER_NAME = "sidecar-log-collector";
+
+ private static final String TESTING_TEMPLATE_FILE_NAME = "testing-pod-template.yaml";
+
+ public static String getPodTemplatePath() {
+ final URL podTemplateUrl =
+ KubernetesPodTemplateTestUtils.class
+ .getClassLoader()
+ .getResource(TESTING_TEMPLATE_FILE_NAME);
+ assertThat(podTemplateUrl, not(nullValue()));
+ return podTemplateUrl.getPath();
+ }
+
+ public static Container getInitContainerInPodTemplate() {
+ return new ContainerBuilder()
+ .withName("artifacts-fetcher")
+ .withImage("testing-init-image")
+ .withCommand(
+ "wget",
+ "https://path/of/StateMachineExample.jar",
+ "-O",
+ "/flink-artifact/myjob.jar")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-artifact")
+ .withMountPath("/flink-artifact")
+ .build())
+ .build();
+ }
+
+ public static Container getSideCarContainerInPodTemplate() {
+ return new ContainerBuilder()
+ .withName("sidecar-log-collector")
+ .withImage("test-sidecar-image")
+ .withCommand("command-to-upload", "/flink-logs/jobmanager.log")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-logs")
+ .withMountPath("/flink-logs")
+ .build())
+ .build();
+ }
+
+ public static Volume[] getVolumesInPodTemplate() {
+ return new Volume[] {
+ new VolumeBuilder()
+ .withName("flink-volume-hostpath")
+ .withNewHostPath()
+ .withNewPath("/tmp")
+ .withType("Directory")
+ .endHostPath()
+ .build(),
+ new VolumeBuilder().withName("flink-artifact").withNewEmptyDir().endEmptyDir().build(),
+ new VolumeBuilder().withName("flink-logs").withNewEmptyDir().endEmptyDir().build()
+ };
+ }
+
+ public static VolumeMount[] getVolumeMountInPodTemplate() {
Review comment:
Same with these names. The `InPodTemplate` does not make a lot of sense if we look at these methods on their own.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/utils/KubernetesUtilsTest.java
##########
@@ -75,6 +81,91 @@ public void testCheckWithFixedPort() {
testCheckAndUpdatePortConfigOption("6123", "16123", "6123");
}
+ @Test
+ public void testLoadPodFromTemplateWithNonExistPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ "/path/of/non-exist.yaml",
+ KubernetesPodTemplateTestUtils.TESTING_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNoMainContainerShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesPodTemplateTestUtils.getPodTemplatePath(),
+ "nonExistMainContainer");
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ assertThat(flinkPod.getPod().getSpec().getContainers().size(), is(2));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckMetaData() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesPodTemplateTestUtils.getPodTemplatePath(),
+ KubernetesPodTemplateTestUtils.TESTING_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getMetadata().getName(), is("pod-template"));
Review comment:
It is not really clear to my why the metadata name is `pod-template` if I only look at this test.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/utils/KubernetesUtilsTest.java
##########
@@ -75,6 +81,91 @@ public void testCheckWithFixedPort() {
testCheckAndUpdatePortConfigOption("6123", "16123", "6123");
}
+ @Test
+ public void testLoadPodFromTemplateWithNonExistPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ "/path/of/non-exist.yaml",
+ KubernetesPodTemplateTestUtils.TESTING_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNoMainContainerShouldReturnEmptyPod() {
Review comment:
`testLoadPodFromTemplateWithNoMainContainerShouldReturnEmptyMainContainer`
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
+
+ private static final String ENTRY_POINT_CLASS =
+ KubernetesSessionClusterEntrypoint.class.getCanonicalName();
+ private static final int RESOURCE_MEMORY = 1456;
+
+ /**
+ * The result pod variables should be set via {@link #setResultPod()} by implementation class.
+ */
+ protected ObjectMeta resultPodMeta;
+
+ protected PodSpec resultPodSpec;
Review comment:
This is quite an implicit contract which is documented via a comment. I am not saying that need to change it but this is in general brittle.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -339,6 +339,11 @@ public void close() {
this.internalClient.close();
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(String file) {
+ return new KubernetesPod(this.internalClient.pods().load(file).get());
Review comment:
Can `get()` return `null`?
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesPodTemplateTestUtils.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ContainerBuilder;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import io.fabric8.kubernetes.api.model.Volume;
+import io.fabric8.kubernetes.api.model.VolumeBuilder;
+import io.fabric8.kubernetes.api.model.VolumeMount;
+import io.fabric8.kubernetes.api.model.VolumeMountBuilder;
+
+import java.net.URL;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.junit.Assert.assertThat;
+
+/** Utilities for the Kubernetes pod template tests. */
+public class KubernetesPodTemplateTestUtils {
+
+ public static final String TESTING_MAIN_CONTAINER_NAME = "testing-main-container";
+ public static final String TESTING_SIDE_CAR_CONTAINER_NAME = "sidecar-log-collector";
+
+ private static final String TESTING_TEMPLATE_FILE_NAME = "testing-pod-template.yaml";
+
+ public static String getPodTemplatePath() {
+ final URL podTemplateUrl =
+ KubernetesPodTemplateTestUtils.class
+ .getClassLoader()
+ .getResource(TESTING_TEMPLATE_FILE_NAME);
+ assertThat(podTemplateUrl, not(nullValue()));
+ return podTemplateUrl.getPath();
+ }
+
+ public static Container getInitContainerInPodTemplate() {
Review comment:
Maybe add a comment that these are the containers specified in the `test-resources/testing-pod-template.yaml`. Maybe this should go to the class JavaDocs.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, String path, String mainContainerName) {
+ if (!new File(path).exists()) {
+ return new FlinkPod.Builder().build();
Review comment:
I am actually wondering whether we shouldn't rather fail hard because the user has specified a pod template which does not exist. The user might be surprised if he has specified the pod template but it is not respected in the end.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/TestingFlinkKubeClient.java
##########
@@ -197,6 +197,11 @@ public void close() {
closeConsumer.accept(null);
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(String file) {
+ return new KubernetesPod(new FlinkPod.Builder().build().getPod());
+ }
Review comment:
Does any of our tests call this method? If not, then maybe throwing an `UnsupportedOperationException` might be better. My concern is that a future user of the `TestingFlinkKubeClient` who wants to load a template file might not see this implementation and wonder why it only creates a default `KubernetesPod`.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/utils/KubernetesUtilsTest.java
##########
@@ -75,6 +81,91 @@ public void testCheckWithFixedPort() {
testCheckAndUpdatePortConfigOption("6123", "16123", "6123");
}
+ @Test
+ public void testLoadPodFromTemplateWithNonExistPathShouldReturnEmptyPod() {
Review comment:
Maybe it is better to fail hard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 145f891faab2190d83a5ea35ac86a025605b43bd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976)
* 6b93e4c21b1edf4cf9c071d88b78ab4230fbc004 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982)
* 04a3a8996bc66bb2d953d5bee380da1a5eff8eac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8d2fd246e442fe92285103908ea6595c64d1c5c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765023075
Thanks @wangyang0918,
This works great, I tested it on my app, and its fine. I have 2 questions:
- With this in place, you now have 2 ways for setting annotations, labels, env, resources and tolerations, etc. You can set them using dedicated decorators or through port template yaml., for example:
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/port: "3000"
prometheus.io/scrape: "true"
labels:
deploy: test
version: alphav1
name: pod-template
spec:
containers:
- name: flink-job-manager
env:
- name: "variable1"
value: "value1"
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "2"
memory: 4Gi
volumeMounts:
- mountPath: /opt/flink/conf
name: flink-config-volume
- mountPath: /mnt/flink/storage
name: pvc-checkpointing
- mountPath: /opt/pvcclaim/test/path1
name: pvc-mount2
subPath: path1
- mountPath: /opt/pvcclaim/test/path2
name: pvc-mount2
subPath: path2
- mountPath: /emptydirclaim
name: edir-mount
- mountPath: /var/local/hp
name: hp-mount
- mountPath: /opt/secret
name: secret-mount
- mountPath: /opt/configmap
name: test-configmap
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: cloudflowflink-token-dbnvh
readOnly: true
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
items:
- key: logback-console.xml
path: logback-console.xml
- key: log4j-console.properties
path: log4j-console.properties
- key: flink-conf.yaml
path: flink-conf.yaml
name: flink-config-testapp
name: flink-config-volume
- name: pvc-checkpointing
persistentVolumeClaim:
claimName: testclaim
- name: pvc-mount2
persistentVolumeClaim:
claimName: testclaim1
- emptyDir: {}
name: edir-mount
- hostPath:
path: /var/local/hp
type: DirectoryOrCreate
name: hp-mount
- name: secret-mount
secret:
defaultMode: 420
items:
- key: token
path: value
secretName: test-secret2
- configMap:
defaultMode: 420
items:
- key: key1
path: "1"
name: test-configmap
name: test-configmap
- name: cloudflowflink-token-dbnvh
secret:
defaultMode: 420
secretName: cloudflowflink-token-dbnvh
This template is setting a lot of parameters, that can be used to set a lot of things. Does it make sense to use template as a main configuration approach and remove other options.
- Both this and owner manager PRs are still not merged. - do you have a time frame for merging/releasing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773801799
@MiLk Thanks for your feedback.
I have added a new commit for the documentation. Now it is ready for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eeea5dc67837785f54f7e18151095f2693a7d391 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782)
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623)
* 021dd8790e62582df174355db6ad1ad978230c65 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631)
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04a3a8996bc66bb2d953d5bee380da1a5eff8eac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993)
* d191789d3ed2e8715fba7054bd98e773f0a28fde Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 021dd8790e62582df174355db6ad1ad978230c65 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631)
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765482738
Thanks @wangyang0918,
I did fix owner reference PR
As for this one, its still up to the user to decide how he wants to set certain parameters - via template or via additional parameters - Flink configuration. I think the best that we can do is to create a table (that you are working on) to specify what can be set via different mechanisms and the precedence rules, for example, if I set a label foo with the value bar in template and then set the same label with the value foo-bar in configuration, the value of the label will be foo-bar, etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r580737573
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
Review comment:
```suggestion
Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
```
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
Review comment:
```suggestion
Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
```
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##########
@@ -367,6 +369,50 @@
code("FlinkKubeClient#checkAndUpdateConfigMap"))
.build());
+ public static final ConfigOption<String> JOB_MANAGER_POD_TEMPLATE =
+ key("kubernetes.pod-template-file.jobmanager")
+ .stringType()
+ .noDefaultValue()
+ .withFallbackKeys(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .withDescription(
+ "Specify a local file that contains the jobmanager pod template definition. "
+ + "It will be used to initialize the jobmanager pod. "
+ + "If not explicitly configured, config option '"
+ + KUBERNETES_POD_TEMPLATE_FILE_KEY
+ + "' will be used.");
+
+ public static final ConfigOption<String> TASK_MANAGER_POD_TEMPLATE =
+ key("kubernetes.pod-template-file.taskmanager")
+ .stringType()
+ .noDefaultValue()
+ .withFallbackKeys(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .withDescription(
+ "Specify a local file that contains the taskmanager pod template definition. "
+ + "It will be used to initialize the taskmanager pod. "
+ + "If not explicitly configured, config option '"
+ + KUBERNETES_POD_TEMPLATE_FILE_KEY
+ + "' will be used.");
+
+ /**
+ * This option is here only for documentation generation, it is the fallback key of
+ * JOB_MANAGER_POD_TEMPLATE and TASK_MANAGER_POD_TEMPLATE.
+ */
+ @SuppressWarnings("unused")
+ public static final ConfigOption<String> KUBERNETES_POD_TEMPLATE =
+ key(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "Specify a local file that contains the pod template definition. "
+ + "It will be used to initialize the jobmanager and taskmanager pod. "
+ + "The main container should be defined with name '"
+ + Constants.MAIN_CONTAINER_NAME
Review comment:
Let's also mention the main container name for `JOB_MANAGER_POD_TEMPLATE ` and `TASK_MANAGER_POD_TEMPLATE` descriptions.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template and not listed in the tables will be unaffected.
Review comment:
```suggestion
All the fields defined in the pod template that are not listed in the tables will be unaffected.
```
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template and not listed in the tables will be unaffected.
+
+Please note that some new config options may be introduced to overwrite more Kubernetes fields,
+your pod template may not work in the future Flink releases.
+We could guarantee that `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`,
+`env`, `init container`, `sidecar container`, `volumes` from pod template will always be respected.
+
+#### Pod Metadata
Review comment:
Currently, the headings `Pod Metadata`, `Pod Spec`, `Container Spec` and `Pod Template Definition` are in the same level.
I would suggest to have a section `Fields Overwritten by Flink` to discuss the overwritten issue. We can move the three tables as well as some texts from the first two paragraphs related to fields overwriting to this section
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template and not listed in the tables will be unaffected.
+
+Please note that some new config options may be introduced to overwrite more Kubernetes fields,
+your pod template may not work in the future Flink releases.
+We could guarantee that `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`,
+`env`, `init container`, `sidecar container`, `volumes` from pod template will always be respected.
Review comment:
```suggestion
Please note that Flink may overwrite more fields from pod templates in future releases.
While it is undetermined which fields might be overwritten in future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
```
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template and not listed in the tables will be unaffected.
+
+Please note that some new config options may be introduced to overwrite more Kubernetes fields,
+your pod template may not work in the future Flink releases.
+We could guarantee that `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`,
+`env`, `init container`, `sidecar container`, `volumes` from pod template will always be respected.
+
+#### Pod Metadata
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Pod MetaData Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>See description </td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>See description</td>
+ <td>The owner reference of JobManager pod will be overwritten by deployment.
+ The deployment owner reference could be set by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a>.
+ The owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Add the annotations from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Add the labels from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+#### Pod Spec
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Pod Spec Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Add image pull secrets from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Add the node selectors from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Add the tolerations from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>"always" for JobManager deployment
+ <br>
+ "never" for TaskManager pod</td>
+ <td>The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-service-account">kubernetes.jobmanager.service-account</a> for JobManager,
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-service-account">kubernetes.taskmanager.service-account</a> for TaskManager</td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>See description</td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+#### Container Spec
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Container Spec Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Add environment variables from <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add additional environment variables specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The image will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>imagePullPolicy</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-policy">kubernetes.container.image.pull-policy</a></td>
+ <td>The image pull policy will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>"flink-job-manager" or "flink-task-manager"</td>
+ <td>The container name will be overwritten by flink ("flink-job-manager" for the JobManager container, and "flink-task-manager" for every TaskManager container).</td>
+ </tr>
+ <tr>
+ <td>resources</td>
+ <td>Memory: value of <a href="{{< ref "docs/deployment/config" >}}#jobmanager-memory-process-size">jobmanager.memory.process.size</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size">taskmanager.memory.process.size</a>
+ <br>
+ CPU: value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-cpu">kubernetes.jobmanager.cpu</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-cpu">kubernetes.taskmanager.cpu</a></td>
+ <td>The memory and cpu resources(including requests and limits) will be overwritten by flink. All other resources(e.g. ephemeral-storage) will be retained.</td>
+ </tr>
+ <tr>
+ <td>volumeMounts</td>
+ <td>See description</td>
+ <td>Flink will add some additional internal volume mounts(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+#### Pod Template Definition
+`pod-template.yaml`
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: jobmanager-pod-template
+spec:
+ initContainers:
+ - name: artifacts-fetcher
+ image: artifacts-fetcher:latest
+ # Use wget or other tools to get user jars from remote storage
+ command: [ 'wget', 'https://path/of/StateMachineExample.jar', '-O', '/flink-artifact/myjob.jar' ]
+ volumeMounts:
+ - mountPath: /flink-artifact
+ name: flink-artifact
+ containers:
+ # Do not change the main container name
+ - name: flink-main-container
Review comment:
The name of the main container is a key information. It should be mentioned in the beginning paragraph, where how this feature should be used is described in general.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template and not listed in the tables will be unaffected.
+
+Please note that some new config options may be introduced to overwrite more Kubernetes fields,
+your pod template may not work in the future Flink releases.
+We could guarantee that `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`,
+`env`, `init container`, `sidecar container`, `volumes` from pod template will always be respected.
+
+#### Pod Metadata
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Pod MetaData Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>See description </td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>See description</td>
+ <td>The owner reference of JobManager pod will be overwritten by deployment.
+ The deployment owner reference could be set by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a>.
+ The owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Add the annotations from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Add the labels from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+#### Pod Spec
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Pod Spec Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Add image pull secrets from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Add the node selectors from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Add the tolerations from <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>"always" for JobManager deployment
+ <br>
+ "never" for TaskManager pod</td>
+ <td>The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-service-account">kubernetes.jobmanager.service-account</a> for JobManager,
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-service-account">kubernetes.taskmanager.service-account</a> for TaskManager</td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>See description</td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+#### Container Spec
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Container Spec Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Add environment variables from <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add additional environment variables specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The image will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>imagePullPolicy</td>
+ <td>Value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-policy">kubernetes.container.image.pull-policy</a></td>
+ <td>The image pull policy will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>"flink-job-manager" or "flink-task-manager"</td>
+ <td>The container name will be overwritten by flink ("flink-job-manager" for the JobManager container, and "flink-task-manager" for every TaskManager container).</td>
+ </tr>
+ <tr>
+ <td>resources</td>
+ <td>Memory: value of <a href="{{< ref "docs/deployment/config" >}}#jobmanager-memory-process-size">jobmanager.memory.process.size</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size">taskmanager.memory.process.size</a>
+ <br>
+ CPU: value of <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-cpu">kubernetes.jobmanager.cpu</a>
+ or <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-cpu">kubernetes.taskmanager.cpu</a></td>
+ <td>The memory and cpu resources(including requests and limits) will be overwritten by flink. All other resources(e.g. ephemeral-storage) will be retained.</td>
+ </tr>
+ <tr>
+ <td>volumeMounts</td>
+ <td>See description</td>
+ <td>Flink will add some additional internal volume mounts(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+#### Pod Template Definition
Review comment:
```suggestion
#### Example of Pod Template
```
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
Review comment:
```suggestion
can be overwritten by Flink.
```
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
+will always be overridden by Flink.
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template and not listed in the tables will be unaffected.
+
+Please note that some new config options may be introduced to overwrite more Kubernetes fields,
+your pod template may not work in the future Flink releases.
+We could guarantee that `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`,
+`env`, `init container`, `sidecar container`, `volumes` from pod template will always be respected.
+
+#### Pod Metadata
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Pod MetaData Key</th>
+ <th class="text-left" style="width: 30%">Modified Value</th>
+ <th class="text-left" style="width: 50%">Description</th>
Review comment:
Looking at the table, it seems the fields Flink overwrite can be categorized as follows.
* **Defined by Flink.** Definitions in template won't take effect. Use cannot control the fields
* **Defined by Config Option.** Definitions in template won't take effect. User can control the fields via config options.
* **Overwritten by Flink.** Definitions in template will take effect, with additional contents from Flink.
I would suggest to explain these categories in advance, and make it a column in the tables. Thus, the tables may contain the following columns: key, category, related config option, description. In this way, the first three columns might be more concise and easier to understand, while the last column provides details that may not be necessary but good to know.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,209 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager via pod template file with the advanced features(e.g. volumes, init container, sidecar container, etc.),
+which are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [kubernetes.pod-template-file]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used to initialize the
+JobManager and TaskManager instead of an empty pod. Please note that some fields of the pod template
Review comment:
```suggestion
to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
JobManager and TaskManager. Please note that some fields of the pod template
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773143835
cc @blublinsky @MiLk I have pushed another commit to support merging Flink config options and pod template. Maybe you could have a try on this and share your feedback. I really appreciate for that.
The documentation is on the way. And I believe this PR will be ready for review in this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
}, {
"hash" : "8fd4af27c4e4a62c2ff4207818aa60decbf41934",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13896",
"triggerID" : "8fd4af27c4e4a62c2ff4207818aa60decbf41934",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* a283f5c426e943b90258a061a263733ce2ba62e4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887)
* 8fd4af27c4e4a62c2ff4207818aa60decbf41934 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13896)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759352375
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 5c06cad57cc46b5e84184ae916138cdf5ae027d4 (Wed Jan 13 10:20:45 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eeea5dc67837785f54f7e18151095f2693a7d391 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581608052
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
Review comment:
The only reason why I introduce this abstract class is to avoid duplicated tests for `KubernetesJobManagerFactoryWithPodTemplateTest` and `KubernetesTaskManagerFactoryWithPodTemplateTest`. It is not about setting up the main part.
Maybe I could rename the `setResultPod` to `getResultPod` in the base class and force the implementation to construct the result pod.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r579634054
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesJobManagerFactoryWithPodTemplateTest.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.kubeclient.FlinkPod;
+import org.apache.flink.kubernetes.kubeclient.KubernetesJobManagerSpecification;
+import org.apache.flink.kubernetes.kubeclient.KubernetesJobManagerTestBase;
+import org.apache.flink.kubernetes.utils.Constants;
+import org.apache.flink.kubernetes.utils.KubernetesUtils;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.PodTemplateSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * General tests for the {@link KubernetesJobManagerFactory} with pod template. These tests will
+ * ensure that init container, volumes, sidecar containers from pod template should be kept after
+ * all decorators.
+ */
+public class KubernetesJobManagerFactoryWithPodTemplateTest extends KubernetesJobManagerTestBase {
Review comment:
The two classes `KubernetesJobManagerFactoryWithPodTemplateTest` and `KubernetesTaskManagerFactoryWithPodTemplateTest` are practically identical, except for `onSetup`.
I think they can be deduplicated.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##########
@@ -367,6 +368,46 @@
code("FlinkKubeClient#checkAndUpdateConfigMap"))
.build());
+ public static final ConfigOption<String> JOB_MANAGER_POD_TEMPLATE =
+ key("kubernetes.jobmanager.pod-template-file")
+ .stringType()
+ .noDefaultValue()
+ .withFallbackKeys(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .withDescription(
+ "Specify a local file that contains the jobmanager pod template. It will be used to "
+ + "initialize the jobmanager pod. If not explicitly configured, config option '"
+ + KUBERNETES_POD_TEMPLATE_FILE_KEY
+ + "' will be used.");
+
+ public static final ConfigOption<String> TASK_MANAGER_POD_TEMPLATE =
+ key("kubernetes.taskmanager.pod-template-file")
+ .stringType()
+ .noDefaultValue()
+ .withFallbackKeys(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .withDescription(
+ "Specify a local file that contains the taskmanager pod template. It will be used to "
+ + "initialize the taskmanager pod. If not explicitly configured, config option '"
+ + KUBERNETES_POD_TEMPLATE_FILE_KEY
+ + "' will be used.");
+
+ /**
+ * This option is here only for documentation generation, it is the fallback key of
+ * JOB_MANAGER_POD_TEMPLATE and TASK_MANAGER_POD_TEMPLATE.
+ */
+ @SuppressWarnings("unused")
+ public static final ConfigOption<String> KUBERNETES_POD_TEMPLATE =
+ key(KUBERNETES_POD_TEMPLATE_FILE_KEY)
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
Review comment:
Shall we explain in the configuration option description that the template should use `Constants.MAIN_CONTAINER_NAME` to refer the main container?
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##########
@@ -367,6 +368,46 @@
code("FlinkKubeClient#checkAndUpdateConfigMap"))
.build());
+ public static final ConfigOption<String> JOB_MANAGER_POD_TEMPLATE =
+ key("kubernetes.jobmanager.pod-template-file")
Review comment:
Minor: I'd suggest to align the option keys as:
* kubernetes.pod-template-file
* kubernetes.pod-template-file.jobmanager
* kubernetes.pod-template-file.taskmanager
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecoratorWithPodTemplateTest.java
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.decorators;
+
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptions;
+import org.apache.flink.kubernetes.kubeclient.FlinkPod;
+import org.apache.flink.kubernetes.kubeclient.KubernetesJobManagerTestBase;
+import org.apache.flink.kubernetes.utils.Constants;
+import org.apache.flink.kubernetes.utils.KubernetesUtils;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.EnvVar;
+import io.fabric8.kubernetes.api.model.LocalObjectReference;
+import io.fabric8.kubernetes.api.model.Pod;
+import io.fabric8.kubernetes.api.model.Quantity;
+import io.fabric8.kubernetes.api.model.ResourceRequirements;
+import io.fabric8.kubernetes.api.model.Toleration;
+import org.hamcrest.Matchers;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests of merging and overwriting Kubernetes fields from {@link KubernetesConfigOptions} and pod
+ * template for the {@link InitJobManagerDecorator}.
+ */
+public class InitJobManagerDecoratorWithPodTemplateTest extends KubernetesJobManagerTestBase {
Review comment:
`InitJobManagerDecoratorWithPodTemplateTest` and `InitTaskManagerDecoratorWithPodTemplateTest` should also be deduplicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-767960682
Moreover, you can't use pod template for anything, that is set by Flink decorators
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773427614
@wangyang0918, I did not have a chance to run the code, but looking at the changes, it looks really good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MiLk commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
MiLk commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773664135
I could confirm that I could set both priority class and ephemeral storage requests and limits 🎉
It seems to behave as expected, thank you for the hard work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-784038867
cc @tillrohrmann could you please have a final check and help with merging?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r579053748
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesParameters.java
##########
@@ -107,4 +107,7 @@
* container(s).
*/
List<Map<String, String>> getEnvironmentsFromSecrets();
+
+ /** The pod template file path. */
+ String getPodTemplateFilePath();
Review comment:
We could return `Optional<String>` to force the existence check.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, @Nullable String path, String mainContainerName) {
+ if (path == null || !new File(path).exists()) {
+ return new FlinkPod.Builder().build();
+ }
+
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(path);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (container.getName().equals(mainContainerName)) {
Review comment:
Are we sure that `container.getName()` is non-null?
Maybe `mainContainerName.equals(container.getName())`?
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/utils/KubernetesUtilsTest.java
##########
@@ -75,6 +88,155 @@ public void testCheckWithFixedPort() {
testCheckAndUpdatePortConfigOption("6123", "16123", "6123");
}
+ @Test
+ public void testLoadPodFromTemplateWithNullPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient, null, JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNonExistPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ "/path/of/non-exist.yaml",
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNoMainContainerShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ "nonExistMainContainer");
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ assertThat(flinkPod.getPod().getSpec().getContainers().size(), is(2));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckMetaData() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getMetadata().getName(), is("pod-template"));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckInitContainer() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getSpec().getInitContainers().size(), is(1));
+ Container expectedInitContainer =
+ new ContainerBuilder()
+ .withName("artifacts-fetcher")
+ .withImage("busybox")
+ .withCommand(
+ "wget",
+ "https://path/of/StateMachineExample.jar",
+ "-O",
+ "/flink-artifact/myjob.jar")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-artifact")
+ .withMountPath("/flink-artifact")
+ .build())
+ .build();
+ assertThat(
+ flinkPod.getPod().getSpec().getInitContainers().get(0), is(expectedInitContainer));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckMainContainer() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getMainContainer().getName(), is(JOB_MANAGER_MAIN_CONTAINER_NAME));
+ final VolumeMount[] expectedVolumeMounts =
+ new VolumeMount[] {
+ new VolumeMountBuilder()
+ .withName("flink-volume-hostpath")
+ .withMountPath("/opt/flink/volumes/hostpath")
+ .build(),
+ new VolumeMountBuilder()
+ .withName("flink-artifact")
+ .withMountPath("/opt/flink/artifacts")
+ .build(),
+ new VolumeMountBuilder()
+ .withName("flink-logs")
+ .withMountPath("/opt/flink/log")
+ .build()
+ };
+ assertThat(
+ flinkPod.getMainContainer().getVolumeMounts(),
+ containsInAnyOrder(expectedVolumeMounts));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckSideCarContainer() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getSpec().getContainers().size(), is(1));
+ Container expectedSideCarContainer =
+ new ContainerBuilder()
+ .withName("sidecar-log-collector")
+ .withImage("busybox")
+ .withCommand("command-to-upload", "/flink-logs/jobmanager.log")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-logs")
+ .withMountPath("/flink-logs")
+ .build())
+ .build();
+ assertThat(
+ flinkPod.getPod().getSpec().getContainers().get(0), is(expectedSideCarContainer));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckVolumes() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
Review comment:
I wonder do we really need to introduce two testing template files. Can we introduce one template file and point both JM/TM configuration options to it?
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/decorators/FlinkConfMountDecoratorTest.java
##########
@@ -243,4 +217,65 @@ public void testDecoratedFlinkContainer() {
assertEquals(Constants.FLINK_CONF_VOLUME, volumeMount.getName());
assertEquals(FLINK_CONF_DIR_IN_POD, volumeMount.getMountPath());
}
+
+ @Test
+ public void testConfigMapWithTaskManagerPodTemplate() throws IOException {
+ final String podTemplateData = "taskmanager pod template data";
+ KubernetesTestUtils.createTemporyFile(
+ podTemplateData, flinkConfDir, TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
Review comment:
Deduplicate with `prepareTaskManagerPodTemplateLocalFile`.
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/decorators/FlinkConfMountDecoratorTest.java
##########
@@ -65,6 +68,9 @@ protected void setupFlinkConfig() {
super.setupFlinkConfig();
this.flinkConfig.set(KubernetesConfigOptions.FLINK_CONF_DIR, FLINK_CONF_DIR_IN_POD);
+ this.flinkConfig.set(
+ KubernetesConfigOptions.TASK_MANAGER_POD_TEMPLATE,
+ new File(flinkConfDir, TASK_MANAGER_POD_TEMPLATE_FILE_NAME).getAbsolutePath());
Review comment:
Maybe use an arbitrary temp folder and file name.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManagerDriver.java
##########
@@ -149,7 +157,8 @@ public void deregisterApplication(
final KubernetesTaskManagerParameters parameters =
createKubernetesTaskManagerParameters(taskExecutorProcessSpec);
final KubernetesPod taskManagerPod =
- KubernetesTaskManagerFactory.buildTaskManagerKubernetesPod(parameters);
+ KubernetesTaskManagerFactory.buildTaskManagerKubernetesPod(
+ Preconditions.checkNotNull(taskManagerPodTemplate).clone(), parameters);
Review comment:
I would suggest to wrap the null-checking, cloning and pod building into `buildTaskManagerPodFromTemplate`.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/FlinkPod.java
##########
@@ -48,6 +48,12 @@ public Container getMainContainer() {
return mainContainer;
}
+ public FlinkPod clone() {
Review comment:
`@Override`
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesTaskManagerParameters.java
##########
@@ -99,6 +99,11 @@ public KubernetesTaskManagerParameters(
.orElse(Collections.emptyList());
}
+ @Override
+ public String getPodTemplateFilePath() {
+ return KubernetesUtils.getTaskManagerPodTemplatePathInPod(flinkConfig);
+ }
Review comment:
This implementation for TM is never used.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/FlinkPod.java
##########
@@ -48,6 +48,12 @@ public Container getMainContainer() {
return mainContainer;
}
+ public FlinkPod clone() {
+ return new FlinkPod(
+ new PodBuilder(this.getPod()).build(),
+ new ContainerBuilder(this.getMainContainer()).build());
Review comment:
It's super implicit that `FlinkPod#pod` is guaranteed not containing the main container, and the main container is only added to the pod in `KubernetesJob/TaskManagerFactory`. Maybe we should rename it to `FlinkPod#podWithoutMainContainer`.
It's not introduced by this PR though.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -309,12 +309,15 @@ public static CompletedCheckpointStore createCompletedCheckpointStore(
* @return KubernetesResource requirements.
*/
public static ResourceRequirements getResourceRequirements(
Review comment:
JavaDoc not updated
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesJobManagerFactoryTest.java
##########
@@ -459,4 +464,28 @@ public void testEmptyHadoopConfDirectory() throws IOException {
.getHadoopConfConfigMapName(
CLUSTER_ID))));
}
+
+ @Test
+ public void testCreateJobManagerDeploymentWithPodTemplate() throws Exception {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ KubernetesJobManagerParameters.JOB_MANAGER_MAIN_CONTAINER_NAME);
+
+ final KubernetesJobManagerSpecification kubernetesJobManagerSpecification =
+ KubernetesJobManagerFactory.buildKubernetesJobManagerSpecification(
+ flinkPod, kubernetesJobManagerParameters);
+
+ final PodTemplateSpec resultPod =
+ kubernetesJobManagerSpecification.getDeployment().getSpec().getTemplate();
+ // Pod template has been applied successfully.
+ assertThat(resultPod.getSpec().getInitContainers().size(), is(1));
+ assertThat(resultPod.getSpec().getContainers().size(), is(2));
+ assertThat(
+ resultPod.getSpec().getContainers().stream()
+ .map(Container::getName)
+ .collect(Collectors.toList()),
+ containsInAnyOrder("flink-job-manager", "sidecar-log-collector"));
Review comment:
I think we need to verify that other functions also take effect after going through all the decorators, like what we do in `KubernetesUtilsTest` (metadata, init container, volumes, etc.) and `InitJob/TaskManagerDecoratorWithPodTemplateTest` (labels, annotations, env, etc.).
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/utils/KubernetesUtilsTest.java
##########
@@ -75,6 +88,155 @@ public void testCheckWithFixedPort() {
testCheckAndUpdatePortConfigOption("6123", "16123", "6123");
}
+ @Test
+ public void testLoadPodFromTemplateWithNullPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient, null, JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNonExistPathShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ "/path/of/non-exist.yaml",
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod(), is(EMPTY_POD.getPod()));
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateWithNoMainContainerShouldReturnEmptyPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ "nonExistMainContainer");
+ assertThat(flinkPod.getMainContainer(), is(EMPTY_POD.getMainContainer()));
+ assertThat(flinkPod.getPod().getSpec().getContainers().size(), is(2));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckMetaData() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getMetadata().getName(), is("pod-template"));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckInitContainer() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getSpec().getInitContainers().size(), is(1));
+ Container expectedInitContainer =
+ new ContainerBuilder()
+ .withName("artifacts-fetcher")
+ .withImage("busybox")
+ .withCommand(
+ "wget",
+ "https://path/of/StateMachineExample.jar",
+ "-O",
+ "/flink-artifact/myjob.jar")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-artifact")
+ .withMountPath("/flink-artifact")
+ .build())
+ .build();
+ assertThat(
+ flinkPod.getPod().getSpec().getInitContainers().get(0), is(expectedInitContainer));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckMainContainer() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getMainContainer().getName(), is(JOB_MANAGER_MAIN_CONTAINER_NAME));
+ final VolumeMount[] expectedVolumeMounts =
+ new VolumeMount[] {
+ new VolumeMountBuilder()
+ .withName("flink-volume-hostpath")
+ .withMountPath("/opt/flink/volumes/hostpath")
+ .build(),
+ new VolumeMountBuilder()
+ .withName("flink-artifact")
+ .withMountPath("/opt/flink/artifacts")
+ .build(),
+ new VolumeMountBuilder()
+ .withName("flink-logs")
+ .withMountPath("/opt/flink/log")
+ .build()
+ };
+ assertThat(
+ flinkPod.getMainContainer().getVolumeMounts(),
+ containsInAnyOrder(expectedVolumeMounts));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckSideCarContainer() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
+ assertThat(flinkPod.getPod().getSpec().getContainers().size(), is(1));
+ Container expectedSideCarContainer =
+ new ContainerBuilder()
+ .withName("sidecar-log-collector")
+ .withImage("busybox")
+ .withCommand("command-to-upload", "/flink-logs/jobmanager.log")
+ .withVolumeMounts(
+ new VolumeMountBuilder()
+ .withName("flink-logs")
+ .withMountPath("/flink-logs")
+ .build())
+ .build();
+ assertThat(
+ flinkPod.getPod().getSpec().getContainers().get(0), is(expectedSideCarContainer));
+ }
+
+ @Test
+ public void testLoadPodFromTemplateAndCheckVolumes() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesTestUtils.getJobManagerPodTemplatePath(),
+ JOB_MANAGER_MAIN_CONTAINER_NAME);
Review comment:
That reveals another issue, can a user point both JM/TM options to the same template file? I think that should be a very common demand. We may also provide a common config option in addition to the JM/TM specific options, similar to what we do for `evn.java.opts`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eeea5dc67837785f54f7e18151095f2693a7d391 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782)
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-766512509
I believe your suggestions make sense. For labels, annotations, node-selectors, tolerations and other similar features, Flink config option will add additional results to such fields. They could be combined easily. However, for other Flink internal generated fields(e.g. `namespace`, `podName`, `restartPolicy`, `mem`, `cpu`, etc.), the values from pod template will be always overwritten.
This should be declared clearly in the documentation with few tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584124388
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +405,74 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, File podTemplateFile, String mainContainerName) {
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(podTemplateFile);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.info(
+ "Could not find main container {} in pod template, using empty one to initialize.",
+ mainContainerName);
+ mainContainer = new ContainerBuilder().build();
+ }
+
+ pod.getInternalResource().getSpec().setContainers(otherContainers);
+ return new FlinkPod(pod.getInternalResource(), mainContainer);
+ }
+
+ public static File getTaskManagerPodTemplateFileInPod() {
+ return new File(
+ Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
+ }
+
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
Review comment:
The reason I do not extract this value from `flinkConfig.get(configOption)` because I want to extract them from `KubernetesParameters`. Then we could reuse the pre condition check(e.g. empty string, length, maybe more in the future) and fallback logic(e.g. `getServiceAccount`).
I know the fallback logic could replaced with `KubernetesConfigOptions#withFallbackKeys`. And I will also get this done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c06cad57cc46b5e84184ae916138cdf5ae027d4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982)
* 031b1373b412f9e7be4c3eb944aa02508b1e10d1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c06cad57cc46b5e84184ae916138cdf5ae027d4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-783242021
@xintongsong Thanks for the review. I have addressed the comments and force pushed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 031b1373b412f9e7be4c3eb944aa02508b1e10d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584124388
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +405,74 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, File podTemplateFile, String mainContainerName) {
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(podTemplateFile);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.info(
+ "Could not find main container {} in pod template, using empty one to initialize.",
+ mainContainerName);
+ mainContainer = new ContainerBuilder().build();
+ }
+
+ pod.getInternalResource().getSpec().setContainers(otherContainers);
+ return new FlinkPod(pod.getInternalResource(), mainContainer);
+ }
+
+ public static File getTaskManagerPodTemplateFileInPod() {
+ return new File(
+ Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
+ }
+
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
Review comment:
The reason I do not extract this value from `flinkConfig.get(configOption)` because I want to extract them from `KubernetesParameters`. Then we could reuse the pre condition check(e.g. empty string, length, maybe more in the future) and fallback logic(e.g. `getServiceAccount`).
I know the fallback logic could replaced with `KubernetesConfigOptions#withFallbackKeys`. And I will create a new ticket if necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c06cad57cc46b5e84184ae916138cdf5ae027d4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982)
* 031b1373b412f9e7be4c3eb944aa02508b1e10d1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765097339
@blublinsky Thanks for your testing on the preview version and sharing your thoughts.
> The priority between pod template and Flink config options
This is a very good and important concern. When I try to introduce the pod template, I always have a rule in my mind, "Flink config options have a high priority than pod template". If you could enable this feature(e.g. labels, annotations, tolerations, etc.) via config options, then you should do it.
Flink allows users to define the JobManager and TaskManager via pod template file with the non-common features, which are not supported by Flink [Kubernetes config options](https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes) directly. Please use [kubernetes.jobmanager.pod-template-file](https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes-jobmanager-pod-template-file)
or [kubernetes.taskmanager.pod-template-file](https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes-taskmanager-pod-template-file) to specify a local file that contains pod definition. It will be used to **initialize** the JobManager/TaskManager instead of an empty pod. Please note that some fields of the pod template will always be overridden by Flink. Refer to the following tables for the full list of pod fields that will be overwritten by Flink. _I am now working to provide such tables in this PR._
I know some guys may argue for that. In the pod template design, it is a supplement for some un-common features and provide enough flexibility to use new K8s features. Also I believe for Flink users, they prefer the config option if they could.
> Both this and owner manager PRs are still not merged. - do you have a time frame for merging/releasing them?
I am trying to refine this PR and also provide a thorough documentation so that users could easily use it. For owner reference PR, I already left some comments few days ago, could you please address them before merging?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 031b1373b412f9e7be4c3eb944aa02508b1e10d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990)
* 5426ab123aef03d4710c0fea1237fa014684f372 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5426ab123aef03d4710c0fea1237fa014684f372 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918)
* 145f891faab2190d83a5ea35ac86a025605b43bd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976)
* 6b93e4c21b1edf4cf9c071d88b78ab4230fbc004 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
* 2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673)
* 2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581924392
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManagerDriver.java
##########
@@ -100,6 +104,17 @@ protected void initializeInternal() throws Exception {
kubeClientOpt =
Optional.of(kubeClientFactory.fromConfiguration(flinkConfig, getIoExecutor()));
podsWatchOpt = watchTaskManagerPods();
+ taskManagerPodTemplate =
+ flinkConfig
+ .getOptional(KubernetesConfigOptions.TASK_MANAGER_POD_TEMPLATE)
+ .map(
+ ignore ->
+ KubernetesUtils.loadPodFromTemplateFile(
+ kubeClientOpt.get(),
+ KubernetesUtils.getTaskManagerPodTemplateFileInPod(
+ flinkConfig),
+ Constants.MAIN_CONTAINER_NAME))
+ .orElse(new FlinkPod.Builder().build());
Review comment:
Just a thought. Couldn't we check here whether `Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME` exists? If it exists, then we load it and if not then we use `FlinkPod.Builder().build()`.
Then we would not have to check a configuration option which is not used afterwards. I think at the moment the contract is if `TASK_MANAGER_POD_TEMPLATE` is set, then we will create a `Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME` file. That's why we can check `KubernetesConfigOptions.TASK_MANAGER_POD_TEMPLATE` here in order to tell whether there is a `Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME` or not. This might be easier if we directly check for `Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME`.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -339,6 +342,16 @@ public void close() {
this.internalClient.close();
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(File file) {
+ if (!file.exists()) {
+ throw new FlinkRuntimeException(
+ String.format("Pod template file %s does not exist.", file),
+ new FileNotFoundException());
Review comment:
I think we don't need to instantiate a `FileNotFoundException` here.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -98,7 +103,7 @@ private Container decorateMainContainer(Container container) {
.withImagePullPolicy(kubernetesJobManagerParameters.getImagePullPolicy().name())
.withResources(requirements)
.withPorts(getContainerPorts())
Review comment:
Also here are a couple of options which might be configured via the pod template and later overwritten by the default value of a Flink configuration option. I think this goes in the same direction as what Xintong mentioned before. Understanding which parts can be freely configured via the pod template and which parts are read from `flink-conf.yaml` might be confusing to the user. Not sure whether this is feasible or not but maybe we could log whenever we overwrite a pod template configuration with an explicitly configured value from `flink-conf.yaml`. Also it might help to log the resulting pod template which is used for creating the pod.
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -98,7 +103,7 @@ private Container decorateMainContainer(Container container) {
.withImagePullPolicy(kubernetesJobManagerParameters.getImagePullPolicy().name())
.withResources(requirements)
.withPorts(getContainerPorts())
Review comment:
Here for example, the user might wanna configure some extra ports for queryable state or so.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
Review comment:
Does this mean that if the config option is not specified, then we will take the default value of the `ConfigOption`?
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
Review comment:
"It will be used ~~, instead of an empty pod,~~ to initialize the"
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Pod Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>"always" for JobManager pod and "never" for TaskManager pod.
+ <br>
+ The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a></td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>Overwritten by Flink</td>
+ <td></td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Main Container Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add additional environment variables specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The container image will be overwritten by flink configuration option.</td>
Review comment:
"by Flink"
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
Review comment:
Flink allows users to define the JobManager and TaskManager pods via template files. This allows to support advanced features
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/resources/FlinkPodTest.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.resources;
+
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.kubeclient.FlinkPod;
+import org.apache.flink.kubernetes.utils.KubernetesUtils;
+
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/** Tests for {@link FlinkPod}. */
+public class FlinkPodTest extends KubernetesTestBase {
+
+ @Test
+ public void testCopyFlinkPod() {
+ final FlinkPod flinkPod =
+ KubernetesUtils.loadPodFromTemplateFile(
+ flinkKubeClient,
+ KubernetesPodTemplateTestUtils.getPodTemplateFile(),
+ KubernetesPodTemplateTestUtils.TESTING_MAIN_CONTAINER_NAME);
+ final FlinkPod clonedFlinkPod = flinkPod.copy();
Review comment:
nit: `copiedFlinkPod`
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -60,14 +60,14 @@ public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
new PodBuilder(flinkPod.getPod())
.withApiVersion(API_VERSION)
.editOrNewMetadata()
- .withLabels(kubernetesJobManagerParameters.getLabels())
- .withAnnotations(kubernetesJobManagerParameters.getAnnotations())
+ .addToLabels(kubernetesJobManagerParameters.getLabels())
+ .addToAnnotations(kubernetesJobManagerParameters.getAnnotations())
.endMetadata()
.editOrNewSpec()
.withServiceAccountName(kubernetesJobManagerParameters.getServiceAccount())
Review comment:
Does it make sense to only override the service account name if an explicit service account name has been configured?
Maybe the following precedence could work:
1. Service account specified via in flink-conf.yaml
2. Service account specified in pod template
3. Default value of `KUBERNETES_SERVICE_ACCOUNT`
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
Review comment:
"contains the pod definition" or "contains the pod template".
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
Review comment:
The mechanism for resolving effective field values can be categorized as follows:
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
Review comment:
Overwritten by Flink sounds as if we are replacing the template fields with something. Maybe "Merged with Config Option" describes it a bit better.
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Pod Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>"always" for JobManager pod and "never" for TaskManager pod.
+ <br>
+ The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a></td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>Overwritten by Flink</td>
+ <td></td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Main Container Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add additional environment variables specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The container image will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>imagePullPolicy</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-policy">kubernetes.container.image.pull-policy</a></td>
+ <td>The container image pull policy will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The container name will be overwritten by flink with "flink-main-container".</td>
Review comment:
"by Flink"
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/FlinkPod.java
##########
@@ -48,6 +48,12 @@ public Container getMainContainer() {
return mainContainer;
}
+ public FlinkPod clone() {
+ return new FlinkPod(
+ new PodBuilder(this.getPod()).build(),
+ new ContainerBuilder(this.getMainContainer()).build());
Review comment:
Maybe we could also rename `getPod()` into `getPodWithoutMainContainer()`
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Pod Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>"always" for JobManager pod and "never" for TaskManager pod.
+ <br>
+ The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a></td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>Overwritten by Flink</td>
+ <td></td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Main Container Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add additional environment variables specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The container image will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>imagePullPolicy</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-policy">kubernetes.container.image.pull-policy</a></td>
+ <td>The container image pull policy will be overwritten by flink configuration option.</td>
Review comment:
"by Flink"
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,235 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files, with the advanced features (e.g. volumes, init containers, sidecar containers, etc.)
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains pod definition. It will be used, instead of an empty pod, to initialize the
+JobManager and TaskManager. The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template can be overwritten by Flink.
+The overwritten mechanism can be categorized as follows.
+* **Defined by Flink.** Definitions in the template won't take effect. User cannot control the fields.
+* **Defined by Config Option.** Definitions in the template won't take effect. User can control the fields via config options.
+* **Overwritten by Flink.** Definitions in the template will take effect, with additional contents from Flink.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+<span class="label label-info">Note</span> Flink may overwrite more fields from pod templates in future releases.
+While it is undetermined which fields might be overwritten in the future, Flink provides the following list of fields that are planned to be long-term supported: `annotations`, `labels`, `imagePullSecrets`, `nodeSelector`, `tolerations`, `env`, `init container`, `sidecar container`, `volumes`.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
+ </tr>
+ <tr>
+ <td>annotations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-annotations">kubernetes.jobmanager.annotations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-annotations">kubernetes.taskmanager.annotations</a></td>
+ <td>Flink will add additional annotations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>labels</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-labels">kubernetes.jobmanager.labels</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-labels">kubernetes.taskmanager.labels</a></td>
+ <td>Flink will add additional labels specified by the Flink configuration options.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Pod Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>imagePullSecrets</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-secrets">kubernetes.container.image.pull-secrets</a></td>
+ <td>Flink will add additional pull secrets specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>nodeSelector</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-node-selector">kubernetes.jobmanager.node-selector</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-node-selector">kubernetes.taskmanager.node-selector</a></td>
+ <td>Flink will add additional node selectors specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>tolerations</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-tolerations">kubernetes.jobmanager.tolerations</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-tolerations">kubernetes.taskmanager.tolerations</a></td>
+ <td>Flink will add additional tolerations specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>restartPolicy</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>"always" for JobManager pod and "never" for TaskManager pod.
+ <br>
+ The JobManager pod will always be restarted by deployment. And the TaskManager pod should not be restarted.</td>
+ </tr>
+ <tr>
+ <td>serviceAccount</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-service-account">kubernetes.service-account</a></td>
+ <td>The JobManager deployment and TaskManager pods will be created with the specified service account.</td>
+ </tr>
+ <tr>
+ <td>volumes</td>
+ <td>Overwritten by Flink</td>
+ <td></td>
+ <td>Flink will add some additional internal ConfigMap volumes(e.g. flink-config-volume, hadoop-config-volume) which is necessary for shipping the Flink configuration and hadoop configuration.</td>
+ </tr>
+ </tbody>
+</table>
+
+**Main Container Spec**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>env</td>
+ <td>Overwritten by Flink</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.master.env.{ENV_NAME}</a>
+ <a href="{{< ref "docs/deployment/config" >}}#forwarding-environment-variables">containerized.taskmanager.env.{ENV_NAME}</a></td>
+ <td>Flink will add additional environment variables specified by the Flink configuration options.</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image">kubernetes.container.image</a></td>
+ <td>The container image will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>imagePullPolicy</td>
+ <td>Defined by Config Option</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-container-image-pull-policy">kubernetes.container.image.pull-policy</a></td>
+ <td>The container image pull policy will be overwritten by flink configuration option.</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The container name will be overwritten by flink with "flink-main-container".</td>
+ </tr>
+ <tr>
+ <td>resources</td>
+ <td>Defined by Config Option</td>
+ <td>Memory: <br>
+ <a href="{{< ref "docs/deployment/config" >}}#jobmanager-memory-process-size">jobmanager.memory.process.size</a>
+ <a href="{{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size">taskmanager.memory.process.size</a>
+ <br>
+ CPU: <br>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-cpu">kubernetes.jobmanager.cpu</a>
+ <a href="{{< ref "docs/deployment/config" >}}#kubernetes-taskmanager-cpu">kubernetes.taskmanager.cpu</a></td>
+ <td>The memory and cpu resources(including requests and limits) will be overwritten by flink. All other resources(e.g. ephemeral-storage) will be retained.</td>
Review comment:
"by Flink"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885)
* a283f5c426e943b90258a061a263733ce2ba62e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584124388
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +405,74 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, File podTemplateFile, String mainContainerName) {
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(podTemplateFile);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.info(
+ "Could not find main container {} in pod template, using empty one to initialize.",
+ mainContainerName);
+ mainContainer = new ContainerBuilder().build();
+ }
+
+ pod.getInternalResource().getSpec().setContainers(otherContainers);
+ return new FlinkPod(pod.getInternalResource(), mainContainer);
+ }
+
+ public static File getTaskManagerPodTemplateFileInPod() {
+ return new File(
+ Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
+ }
+
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
Review comment:
The reason I do not extract this value from `flinkConfig.get(configOption)` because I want to extract them from `KubernetesParameters`. Then we could reuse the pre condition check(e.g. empty string, length, maybe more in the future) and fallback logic(e.g. `getServiceAccount`).
I know the fallback logic could replaced with `KubernetesConfigOptions#withFallbackKeys`. And I will add a new commit for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02960c66626c4bc55cd2bdd3a10406eb090eb412 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824)
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885)
* a283f5c426e943b90258a061a263733ce2ba62e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787803898
Make sense to me. I will create another ticket and PR to update the current application e2e test to also verify pod template.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623)
* 021dd8790e62582df174355db6ad1ad978230c65 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631)
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-767917049
@wangyang0918
Did some more experiments with this PR and took more careful look at the code.
In the InitJob/TaskManagerDecorator, editOrNewMetadata() overwrites all of the information from the pod template. As a result, it looks like any metadata settings in the template will be wiped out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680)
* eeea5dc67837785f54f7e18151095f2693a7d391 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584529114
##########
File path: docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
##########
@@ -324,4 +324,240 @@ $ kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit
Please refer to the official Kubernetes documentation on [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for more information.
+### Pod Template
+
+Flink allows users to define the JobManager and TaskManager pods via template files. This allows to support advanced features
+that are not supported by Flink [Kubernetes config options]({{< ref "docs/deployment/config" >}}#kubernetes) directly.
+Use [`kubernetes.pod-template-file`]({{< ref "docs/deployment/config" >}}#kubernetes-pod-template-file)
+to specify a local file that contains the pod definition. It will be used to initialize the JobManager and TaskManager.
+The main container should be defined with name `flink-main-container`.
+Please refer to the [pod template example](#example-of-pod-template) for more information.
+
+#### Fields Overwritten by Flink
+
+Some fields of the pod template will be overwritten by Flink.
+The mechanism for resolving effective field values can be categorized as follows:
+* **Defined by Flink:** User cannot configure it.
+* **Defined by the user:** User can freely specify this value. Flink framework won't set any additional values and the effective value derives from the config option and the template.
+
+ Precedence order: First an explicit config option value is taken, then the value in pod template and at last the default value of a config option if nothing is specified.
+* **Merged with Flink:** Flink will merge values for a setting with a user defined value (see precedence order for "Defined by the user"). Flink values have precedence in case of same name fields.
+
+Refer to the following tables for the full list of pod fields that will be overwritten.
+All the fields defined in the pod template that are not listed in the tables will be unaffected.
+
+**Pod Metadata**
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 10%">Key</th>
+ <th class="text-left" style="width: 20%">Category</th>
+ <th class="text-left" style="width: 30%">Related Config Options</th>
+ <th class="text-left" style="width: 40%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>name</td>
+ <td>Defined by Flink</td>
+ <td></td>
+ <td>The JobManager pod name will be overwritten with the deployment which is defined by <a href="{{< ref "docs/deployment/config" >}}#kubernetes-cluster-id">kubernetes.cluster-id</a>.
+ The TaskManager pod names will be overwritten with the pattern <code><clusterID>-<attempt>-<index></code> which is generated by Flink ResourceManager.</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-namespace">kubernetes.namespace</a></td>
+ <td>Both the JobManager deployment and TaskManager pods will be created in the specified namespace.</td>
+ </tr>
+ <tr>
+ <td>ownerReferences</td>
+ <td>Defined by the user</td>
+ <td><a href="{{< ref "docs/deployment/config" >}}#kubernetes-jobmanager-owner-reference">kubernetes.jobmanager.owner.reference</a></td>
+ <td>The owner reference of JobManager deployment could be set by configuration option.
+ And the owner reference of every TaskManager pod will be set to JobManager deployment.</td>
Review comment:
Sounds good to make these values then "defined by Flink".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r582190272
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/FlinkConfMountDecorator.java
##########
@@ -150,6 +173,24 @@ private Pod decoratePod(Pod pod) {
return Collections.singletonList(flinkConfConfigMap);
}
+ private Optional<File> getTaskManagerPodTemplateFile() {
+ return kubernetesComponentConf
+ .getFlinkConfiguration()
+ .getOptional(KubernetesConfigOptions.TASK_MANAGER_POD_TEMPLATE)
+ .map(
+ file -> {
+ final File podTemplateFile = new File(file);
+ if (!podTemplateFile.exists()) {
+ throw new FlinkRuntimeException(
+ new FileNotFoundException(
+ String.format(
+ "Pod template file %s does not exist.",
+ file)));
+ }
+ return podTemplateFile;
+ });
Review comment:
I think this breaks currently. If the user starts a cluster with `kubernetes.pod-template-file` pointing to a local file, then the started JobManager pod won't have this file but still tries to access it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 removed a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 removed a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787086316
cc @tillrohrmann comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-768026710
> For labels, annotations, node-selectors, tolerations and other similar features, Flink config option will add additional results to such fields. They could be combined easily. However, for other Flink internal generated fields(e.g. namespace, podName, restartPolicy, mem, cpu, etc.), the values from pod template will be always overwritten.
> As a result, it looks like any metadata settings in the template will be wiped out
@blublinsky Yes, this is what I am changing to get your suggestion integrated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584529334
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitJobManagerDecorator.java
##########
@@ -49,64 +50,104 @@
public class InitJobManagerDecorator extends AbstractKubernetesStepDecorator {
private final KubernetesJobManagerParameters kubernetesJobManagerParameters;
+ private final Configuration flinkConfig;
public InitJobManagerDecorator(KubernetesJobManagerParameters kubernetesJobManagerParameters) {
this.kubernetesJobManagerParameters = checkNotNull(kubernetesJobManagerParameters);
+ this.flinkConfig = checkNotNull(kubernetesJobManagerParameters.getFlinkConfiguration());
}
@Override
public FlinkPod decorateFlinkPod(FlinkPod flinkPod) {
- final Pod basicPod =
- new PodBuilder(flinkPod.getPod())
- .withApiVersion(API_VERSION)
- .editOrNewMetadata()
- .withLabels(kubernetesJobManagerParameters.getLabels())
- .withAnnotations(kubernetesJobManagerParameters.getAnnotations())
- .endMetadata()
- .editOrNewSpec()
- .withServiceAccountName(kubernetesJobManagerParameters.getServiceAccount())
- .withImagePullSecrets(kubernetesJobManagerParameters.getImagePullSecrets())
- .withNodeSelector(kubernetesJobManagerParameters.getNodeSelector())
- .withTolerations(
- kubernetesJobManagerParameters.getTolerations().stream()
- .map(
- e ->
- KubernetesToleration.fromMap(e)
- .getInternalResource())
- .collect(Collectors.toList()))
- .endSpec()
- .build();
+ final PodBuilder basicPodBuilder = new PodBuilder(flinkPod.getPodWithoutMainContainer());
+
+ // Overwrite fields
+ final String serviceAccountName =
+ KubernetesUtils.resolveUserDefinedValue(
+ flinkConfig,
+ KubernetesConfigOptions.JOB_MANAGER_SERVICE_ACCOUNT,
+ kubernetesJobManagerParameters.getServiceAccount(),
+ flinkPod.getPodWithoutMainContainer().getSpec().getServiceAccount(),
+ "service account name");
+ if (flinkPod.getPodWithoutMainContainer().getSpec().getRestartPolicy() != null) {
+ logger.info(
+ "The restart policy of JobManager pod will be overwritten to 'always' "
+ + "since it is controlled by the Kubernetes deployment.");
Review comment:
Alright, thanks for the clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787086316
cc @tillrohrmann comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787803898
Make sense to me. I have created the ticket FLINK-21541 to track this follow-up and will get it done soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584550780
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -431,6 +435,61 @@ public static File getTaskManagerPodTemplateFileInPod() {
Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
}
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
+ @Nullable String valueOfPodTemplate,
+ String fieldDescription) {
+ final String resolvedValue;
+ if (valueOfPodTemplate != null) {
+ // The config option is explicitly set.
+ if (flinkConfig.contains(configOption)) {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ LOG.info(
+ "The {} configured in pod template will be overwritten to '{}' "
+ + "because of explicitly configured options.",
+ fieldDescription,
+ resolvedValue);
+ } else {
+ resolvedValue = valueOfPodTemplate;
+ }
+ } else {
+ resolvedValue = valueOfConfigOptionOrDefault;
+ }
+ return resolvedValue;
+ }
+
+ /**
+ * Get the service account from the input pod first, if not specified, the service account name
+ * will be used.
Review comment:
What is the difference between service account and service account name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e37e897e9cce7d900d4d22a3e50e1314f5239cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639)
* 2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673)
* 2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581600918
##########
File path: flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/factory/KubernetesFactoryWithPodTemplateTestBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.kubeclient.factory;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.kubernetes.KubernetesPodTemplateTestUtils;
+import org.apache.flink.kubernetes.KubernetesTestBase;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint;
+import org.apache.flink.kubernetes.utils.Constants;
+
+import io.fabric8.kubernetes.api.model.Container;
+import io.fabric8.kubernetes.api.model.ObjectMeta;
+import io.fabric8.kubernetes.api.model.PodSpec;
+import org.junit.Test;
+
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test base for the {@link KubernetesJobManagerFactory} and {@link KubernetesTaskManagerFactory}
+ * with pod template. These tests will ensure that annotations, labels, imagePullSecrets,
+ * nodeSelector, tolerations, env, init container, sidecar container, volumes from pod template
+ * should be kept after all decorators.
+ */
+public abstract class KubernetesFactoryWithPodTemplateTestBase extends KubernetesTestBase {
+
+ private static final String ENTRY_POINT_CLASS =
+ KubernetesSessionClusterEntrypoint.class.getCanonicalName();
+ private static final int RESOURCE_MEMORY = 1456;
+
+ /**
+ * The result pod variables should be set via {@link #setResultPod()} by implementation class.
+ */
+ protected ObjectMeta resultPodMeta;
+
+ protected PodSpec resultPodSpec;
Review comment:
Since `setResultPod` is an abstract method, then we will force the implementors to set the result pod. Moreover, I will add a null check in the `onSetup` after `setResultPod`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787798846
Thanks for updating the PR @wangyang0918. The changes look good to me. Merging this PR now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-783980838
@xintongsong Thanks for reviewing the documentation. I have integrated your comments. Please have another look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581591845
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -339,6 +339,11 @@ public void close() {
this.internalClient.close();
}
+ @Override
+ public KubernetesPod loadPodFromTemplateFile(String file) {
+ return new KubernetesPod(this.internalClient.pods().load(file).get());
Review comment:
If the `file` is an invalid file path, the fabric8 Kubernetes client will fail with exception. Given that this is uncertain assumption, I will move the existence check from `KubernetesUtils#loadPodFromTemplateFile` to here.
The `load()#get()` will always return a non-null value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787426984
cc @tillrohrmann Comments addressed. Please have a look at your convenience.
I have also changed a bit about the service account because of the following reasons.
* The Kubernetes provides two ways to set the service account(`withServiceAccount` and `withServiceAccountName`). We should set them both.
* When checking whether config options are explicitly set, the fallback config option should also be checked. So I am introducing the `withFallbackKeys`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
}, {
"hash" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13639",
"triggerID" : "4e37e897e9cce7d900d4d22a3e50e1314f5239cb",
"triggerType" : "PUSH"
}, {
"hash" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13673",
"triggerID" : "2d9c758d0a30150d11aa9b0cb9bd4f48aba548c8",
"triggerType" : "PUSH"
}, {
"hash" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13680",
"triggerID" : "2c0d19f8e036530c8e6f2d2b83e33e2f8ee24164",
"triggerType" : "PUSH"
}, {
"hash" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13782",
"triggerID" : "eeea5dc67837785f54f7e18151095f2693a7d391",
"triggerType" : "PUSH"
}, {
"hash" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13824",
"triggerID" : "02960c66626c4bc55cd2bdd3a10406eb090eb412",
"triggerType" : "PUSH"
}, {
"hash" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d3fb87bb54b433d0ac3489edce397b7a9e8cb51",
"triggerType" : "PUSH"
}, {
"hash" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13885",
"triggerID" : "cc9b29c05ee0a3ed5d6f8c2c67cbf47c39f2cb0c",
"triggerType" : "PUSH"
}, {
"hash" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887",
"triggerID" : "a283f5c426e943b90258a061a263733ce2ba62e4",
"triggerType" : "PUSH"
}, {
"hash" : "8fd4af27c4e4a62c2ff4207818aa60decbf41934",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8fd4af27c4e4a62c2ff4207818aa60decbf41934",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d3fb87bb54b433d0ac3489edce397b7a9e8cb51 UNKNOWN
* a283f5c426e943b90258a061a263733ce2ba62e4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13887)
* 8fd4af27c4e4a62c2ff4207818aa60decbf41934 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13523",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
}, {
"hash" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573",
"triggerID" : "cc7552c61220c0b06e433c0294f27a05a3d80ef6",
"triggerType" : "PUSH"
}, {
"hash" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623",
"triggerID" : "4b766f27d466606f4c9f5b611434e17c3a20dc44",
"triggerType" : "PUSH"
}, {
"hash" : "021dd8790e62582df174355db6ad1ad978230c65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631",
"triggerID" : "021dd8790e62582df174355db6ad1ad978230c65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc7552c61220c0b06e433c0294f27a05a3d80ef6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13573)
* 4b766f27d466606f4c9f5b611434e17c3a20dc44 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13623)
* 021dd8790e62582df174355db6ad1ad978230c65 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13631)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 031b1373b412f9e7be4c3eb944aa02508b1e10d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990)
* 5426ab123aef03d4710c0fea1237fa014684f372 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r584529619
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +405,74 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, File podTemplateFile, String mainContainerName) {
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(podTemplateFile);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.info(
+ "Could not find main container {} in pod template, using empty one to initialize.",
+ mainContainerName);
+ mainContainer = new ContainerBuilder().build();
+ }
+
+ pod.getInternalResource().getSpec().setContainers(otherContainers);
+ return new FlinkPod(pod.getInternalResource(), mainContainer);
+ }
+
+ public static File getTaskManagerPodTemplateFileInPod() {
+ return new File(
+ Constants.POD_TEMPLATE_DIR_IN_POD, Constants.TASK_MANAGER_POD_TEMPLATE_FILE_NAME);
+ }
+
+ /**
+ * Resolve the user defined value with the precedence. First an explicit config option value is
+ * taken, then the value in pod template and at last the default value of a config option if
+ * nothing is specified.
+ *
+ * @param flinkConfig flink configuration
+ * @param configOption the config option to define the Kubernetes fields
+ * @param valueOfConfigOptionOrDefault the value defined by explicit config option or default
+ * @param valueOfPodTemplate the value defined in the pod template
+ * @param fieldDescription Kubernetes fields description
+ * @param <T> The type of value associated with the configuration option.
+ * @return the resolved value
+ */
+ public static <T> String resolveUserDefinedValue(
+ Configuration flinkConfig,
+ ConfigOption<T> configOption,
+ String valueOfConfigOptionOrDefault,
Review comment:
Ah ok. I see. Thanks for the clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MiLk commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
MiLk commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-772208383
No, I'm doing the following:
```
apiVersion: v1
kind: Pod
metadata:
name: pod-template
spec:
containers:
- name: flink-task-manager
resources:
requests:
ephemeral-storage: 256Mi
limits:
ephemeral-storage: 256Mi
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787104579
cc @tillrohrmann Comments addressed. Please have a look at your convenience.
I have also changed a bit about the service account because of following reasons.
* The Kubernetes provides two ways to set the service account(`withServiceAccount` and `withServiceAccountName`)
* When checking whether config options are explicitly set, the fallback config option should also be checked. So I am introducing the `withFallbackKeys`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
blublinsky commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-765482738
Thanks @wangyang0918,
I did fix owner reference PR
As for this one, its still up to the user to decide how he wants to set certain parameters - via template or via additional parameters. I think the best that we can do is to create a table (that you are working on) to specify what can be set via different mechanisms and the precedence rules, for example, if I set a label foo with the value bar in template and then set the same label with the value foobar in configuration, the value of the label will be foobar, etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-787802120
One thing we should probably do as follow up it to add an end-to-end test for the pod template. Maybe we can adapt one of the existing K8s e2e tests. That way we ensure that we don't accidentally break this feature w/o us noticing it (e.g. calling the file existence check for the specified file in the pods). What do you think @wangyang0918?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-773143835
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #14629:
URL: https://github.com/apache/flink/pull/14629#discussion_r581595192
##########
File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##########
@@ -396,6 +401,42 @@ public static String getCommonStartCommand(
.collect(Collectors.toList());
}
+ public static FlinkPod loadPodFromTemplateFile(
+ FlinkKubeClient kubeClient, String path, String mainContainerName) {
+ if (!new File(path).exists()) {
+ return new FlinkPod.Builder().build();
+ }
+
+ final KubernetesPod pod = kubeClient.loadPodFromTemplateFile(path);
+ final List<Container> otherContainers = new ArrayList<>();
+ Container mainContainer = null;
+
+ for (Container container : pod.getInternalResource().getSpec().getContainers()) {
+ if (mainContainerName.equals(container.getName())) {
+ mainContainer = container;
+ } else {
+ otherContainers.add(container);
+ }
+ }
+
+ if (mainContainer == null) {
+ LOG.warn(
Review comment:
Usually, the main container is necessary to set but it is not required.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-759361463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11982",
"triggerID" : "5c06cad57cc46b5e84184ae916138cdf5ae027d4",
"triggerType" : "PUSH"
}, {
"hash" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11990",
"triggerID" : "031b1373b412f9e7be4c3eb944aa02508b1e10d1",
"triggerType" : "PUSH"
}, {
"hash" : "5426ab123aef03d4710c0fea1237fa014684f372",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12918",
"triggerID" : "5426ab123aef03d4710c0fea1237fa014684f372",
"triggerType" : "PUSH"
}, {
"hash" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12976",
"triggerID" : "145f891faab2190d83a5ea35ac86a025605b43bd",
"triggerType" : "PUSH"
}, {
"hash" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12982",
"triggerID" : "6b93e4c21b1edf4cf9c071d88b78ab4230fbc004",
"triggerType" : "PUSH"
}, {
"hash" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12993",
"triggerID" : "04a3a8996bc66bb2d953d5bee380da1a5eff8eac",
"triggerType" : "PUSH"
}, {
"hash" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13043",
"triggerID" : "d191789d3ed2e8715fba7054bd98e773f0a28fde",
"triggerType" : "PUSH"
}, {
"hash" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490",
"triggerID" : "b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259",
"triggerType" : "PUSH"
}, {
"hash" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8d2fd246e442fe92285103908ea6595c64d1c5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7790eb7008ad83fb5f8cc3a24b5e8ce5abfe259 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13490)
* b8d2fd246e442fe92285103908ea6595c64d1c5c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on pull request #14629:
URL: https://github.com/apache/flink/pull/14629#issuecomment-784837661
cc @tillrohrmann I have addressed the comments and force pushed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org