You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/11/28 15:00:53 UTC
[GitHub] [flink-kubernetes-operator] pvary opened a new pull request, #455: FLINK-30199: Add a script to run Kubernetes Operator e2e tests manually
pvary opened a new pull request, #455:
URL: https://github.com/apache/flink-kubernetes-operator/pull/455
## What is the purpose of the change
Adding a possibility to run the e2e tests manually and using this same script to run the tests on the CI
## Brief change log
- Refactored the test steps to `functions` in the util.sh
- Changed the `ci.yml` to use these new fuctions
- Created a shell script to run the tests
## Verifying this change
- This change is already covered by existing e2e - will see the test results
- Also manually executed the test script
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
- Core observer or reconciler logic that is regularly executed: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? There is a help text in the script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mbalassi commented on pull request #455: FLINK-30199: Add a script to run Kubernetes Operator e2e tests manually
Posted by GitBox <gi...@apache.org>.
mbalassi commented on PR #455:
URL: https://github.com/apache/flink-kubernetes-operator/pull/455#issuecomment-1330618530
cc @wangyang0918 for your information @pvary has reworked the test suite to block tests together on the same container runs, we hope this effectively will give us more CI bandwidth. The minikube instances are reused between the tests, the operator itself is always reinstalled to cleanly separate the logs. I am happy to review, just fyi. 😏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] pvary commented on a diff in pull request #455: FLINK-30199: Add a script to run Kubernetes Operator e2e tests manually
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #455:
URL: https://github.com/apache/flink-kubernetes-operator/pull/455#discussion_r1035737221
##########
e2e-tests/utils.sh:
##########
@@ -265,3 +270,75 @@ function create_namespace() {
fi;
}
+
+function install_cert_manager() {
+ if [[ -n ${DEBUG} ]]; then
+ kubectl get pods -A
+ fi
+ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
+ kubectl -n cert-manager wait --all=true --for=condition=Available --timeout=300s deploy
+}
+
+function build_image() {
+ export SHELL=/bin/bash
+ export DOCKER_BUILDKIT=1
+ eval $(minikube docker-env)
+ docker build --progress=plain --no-cache -f ./Dockerfile -t flink-kubernetes-operator:ci-latest --progress plain .
+ if [[ -n ${DEBUG} ]]; then
+ docker images
+ fi
+}
+
+function install_operator() {
+ local namespace=$1
+
+ create_namespace flink
+ if [[ -n ${DEBUG} ]]; then
+ debug="--debug"
+ fi
+ helm ${debug} install flink-kubernetes-operator -n ${namespace} helm/flink-kubernetes-operator --set image.repository=flink-kubernetes-operator --set image.tag=ci-latest --create-namespace --set 'watchNamespaces={default,flink}'
Review Comment:
The namespaces were test parameters even before this change, and the watched namespaces were only an extra param which did not change between the tests.
I tried to keep the existing testing behaviour in this change, and especially do not want to change the tests themselves, so we can be sure that the PR does not cause any unwanted changes.
We might want to do this in a different PR, but even there, I am not sure if it is worth it. I think we gain more by keeping the tests simple and easy to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mbalassi commented on a diff in pull request #455: FLINK-30199: Add a script to run Kubernetes Operator e2e tests manually
Posted by GitBox <gi...@apache.org>.
mbalassi commented on code in PR #455:
URL: https://github.com/apache/flink-kubernetes-operator/pull/455#discussion_r1034729637
##########
e2e-tests/utils.sh:
##########
@@ -265,3 +270,75 @@ function create_namespace() {
fi;
}
+
+function install_cert_manager() {
+ if [[ -n ${DEBUG} ]]; then
+ kubectl get pods -A
+ fi
+ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
+ kubectl -n cert-manager wait --all=true --for=condition=Available --timeout=300s deploy
+}
+
+function build_image() {
+ export SHELL=/bin/bash
+ export DOCKER_BUILDKIT=1
+ eval $(minikube docker-env)
+ docker build --progress=plain --no-cache -f ./Dockerfile -t flink-kubernetes-operator:ci-latest --progress plain .
+ if [[ -n ${DEBUG} ]]; then
+ docker images
+ fi
+}
+
+function install_operator() {
+ local namespace=$1
+
+ create_namespace flink
+ if [[ -n ${DEBUG} ]]; then
+ debug="--debug"
+ fi
+ helm ${debug} install flink-kubernetes-operator -n ${namespace} helm/flink-kubernetes-operator --set image.repository=flink-kubernetes-operator --set image.tag=ci-latest --create-namespace --set 'watchNamespaces={default,flink}'
Review Comment:
I would prefer to be able to configure the watch namespaces too for the local testing use case, having it hardcoded might be surprising.
##########
e2e-tests/run_tests.sh:
##########
@@ -0,0 +1,239 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+ROOT_DIR=$(dirname $(dirname "$(readlink -f "$0")"))
+source "${ROOT_DIR}/e2e-tests/utils.sh"
Review Comment:
When running from the `e2e-tests` directory directly this results in:
```
./run_tests.sh: line 21: ./e2e-tests/utils.sh: No such file or directory
```
##########
e2e-tests/utils.sh:
##########
@@ -265,3 +270,75 @@ function create_namespace() {
fi;
}
+
+function install_cert_manager() {
+ if [[ -n ${DEBUG} ]]; then
+ kubectl get pods -A
+ fi
+ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
+ kubectl -n cert-manager wait --all=true --for=condition=Available --timeout=300s deploy
+}
+
+function build_image() {
+ export SHELL=/bin/bash
+ export DOCKER_BUILDKIT=1
+ eval $(minikube docker-env)
+ docker build --progress=plain --no-cache -f ./Dockerfile -t flink-kubernetes-operator:ci-latest --progress plain .
+ if [[ -n ${DEBUG} ]]; then
+ docker images
+ fi
+}
+
+function install_operator() {
+ local namespace=$1
+
+ create_namespace flink
+ if [[ -n ${DEBUG} ]]; then
+ debug="--debug"
+ fi
+ helm ${debug} install flink-kubernetes-operator -n ${namespace} helm/flink-kubernetes-operator --set image.repository=flink-kubernetes-operator --set image.tag=ci-latest --create-namespace --set 'watchNamespaces={default,flink}'
Review Comment:
@pvary on second thought it is up to you how far you want to take this as this would mean also making the CR namespaces customizable, your call whether you think it is worth it.
##########
e2e-tests/run_tests.sh:
##########
@@ -0,0 +1,239 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+ROOT_DIR=$(dirname $(dirname "$(readlink -f "$0")"))
+source "${ROOT_DIR}/e2e-tests/utils.sh"
Review Comment:
This is due to the `readlink` implementation on my OSX. I missed this error in the log:
`readlink: illegal option -- f`
There are a couple suggestions here:
https://stackoverflow.com/questions/1055671/how-can-i-get-the-behavior-of-gnus-readlink-f-on-a-mac
WDYT @pvary?
##########
e2e-tests/run_tests.sh:
##########
@@ -0,0 +1,239 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+ROOT_DIR=$(dirname $(dirname "$(readlink -f "$0")"))
+source "${ROOT_DIR}/e2e-tests/utils.sh"
+
+function help() {
+ # Display Help
+ echo "Script to run the flink kubernetes operator e2e tests."
+ echo
+ echo "Syntax: run_tests.sh [-i|-f|-m|-n|-s|-c|-h] scripts"
+ echo
+ echo "Options:"
+ echo " -i The Flink image to use (eg. flink:1.16 / flink:1.15 / flink:1.14). Only needed if the default docker image for the Flink version need to be changed."
+ echo " -f The Flink version(s) to use (eg. v1_16 / v1_15 / v1_14). Could contain multiple values separated by ','"
+ echo " -m The run mode(s) to use (eg. native / standalone). Could contain multiple values separated by ','"
+ echo " -n The namespace(s) to install the operator to. Could contain multiple values separated by ','"
+ echo " -s If set then skips the operator build (Make sure that minikube already has a version of the operator with the name of 'flink-kubernetes-operator:ci-latest')."
+ echo " -k After the test run the operator is not uninstalled. Will exit after the first test run."
+ echo " -d If set then the script will print debug logs."
+ echo " -h Print this help."
+ echo
+ echo "On MAC do not forget to start the 'minikube tunnel' in a different terminal."
+ echo "Visit this terminal from time to time, as it will ask you for root password."
+ echo "This is needed to forward the rest endpoint requests to port 80."
+ echo
+ echo "Examples:"
+ echo " Start the test_application_operations.sh (default) test on a newly built and deployed operator."
+ echo " The test will use the default settings for every config value:"
+ echo " ./run_tests.sh"
+ echo
+ echo " Start a single test using the currently compiled docker image of the operator which should be available on the minikube."
+ echo " The operator will be deployed in the 'flink' namespace:"
+ echo " ./run_tests.sh -s -namespace flink"
+ echo
+ echo " Run multiple tests:"
+ echo " ./run_tests.sh test_application_operations.sh test_sessionjob_operations.sh"
+ echo
+ echo " Run a single test and show debug logs:"
+ echo " ./run_tests.sh -d"
+ echo
+ echo " Use every possible configuration:"
+ echo " ./run_tests.sh -i 'flink:1.15' -f 'v1_15' -m standalone -n flink -s -k test_sessionjob_operations.sh"
+ echo
+ echo " Run every possible test configuration:"
+ echo " ./run_tests.sh -f v1_16,v1_15,v1_14,v1_13 -m native,standalone -n flink,default -d test_multi_sessionjob.sh test_application_operations.sh test_application_kubernetes_ha.sh test_sessionjob_kubernetes_ha.sh test_sessionjob_operations.sh"
+ echo
+ echo
+ echo " Use the script to compile and deploy the current version of the operator, and check the deployment:"
+ echo " ./run_tests.sh -k"
+ echo
+}
+
+# Default values
+image=""
+flink_versions="v1_16"
+modes="native"
+namespaces="default"
+skip_operator="false"
+keep_operator="false"
+
+# Flink version to Flink docker image mapping. Add new version and image here if it is supported.
+versions=("v1_16" "v1_15" "v1_14" "v1_13")
+images=("flink:1.16" "flink:1.15" "flink:1.14" "flink:1.13")
+
+while getopts "i:f:m:n:hsdk" option; do
+ case $option in
+ i) image=$OPTARG;;
+ f) flink_versions=$OPTARG;;
+ m) modes=$OPTARG;;
+ n) namespaces=$OPTARG;;
+ s) skip_operator="true";;
+ k) keep_operator="true";;
+ d) DEBUG="true";;
+ h) # display Help
+ help
+ exit;;
+ \?) # Invalid option
+ echo "Error: Invalid option"
+ exit;;
+ esac
+done
+
+scripts=${@:$OPTIND}
+if [[ -z "${scripts}" ]]; then
+ scripts="test_application_operations.sh"
+fi
+
+echo
+echo "-----------------------------------"
+echo "Start testing with the following parameters:"
+echo " Image: ${image}"
+echo " Flink version(s): ${flink_versions}"
+echo " Mode(s): ${modes}"
+echo " Operator namespace(s): ${namespaces}"
+echo " Skip operator build: ${skip_operator}"
+echo " Script() to run: ${scripts}"
+echo "-----------------------------------"
+
+pushd $ROOT_DIR
+echo
+echo "-----------------------------------"
+echo "Starting minikube"
+echo "-----------------------------------"
+echo
+time start_minikube
+
+echo
+echo "-----------------------------------"
+echo "Installing certificate manager"
+echo "-----------------------------------"
+echo
+time install_cert_manager
+
+if [[ "$skip_operator" = true ]]; then
+ echo
+ echo "-----------------------------------"
+ echo "Skipping operator build"
+ echo "-----------------------------------"
+else
+ echo
+ echo "-----------------------------------"
+ echo "Building operator"
+ echo "-----------------------------------"
+ echo
+ time build_image
+fi
+
+if [[ ! -z $(get_operator_pod_name 2>/dev/null) ]]; then
+ operator_pod_namespace=$(get_operator_pod_namespace)
+ echo
+ echo "-----------------------------------"
+ echo "Found existing operator. Please remove and restart the script. The following commands could be used:"
+ echo " helm delete flink-kubernetes-operator -n ${operator_pod_namespace}"
+ echo " kubectl delete namespace flink"
+ echo " kubectl delete serviceaccount flink"
+ echo " kubectl delete role flink"
+ echo " kubectl delete rolebinding flink-role-binding"
+ echo "-----------------------------------"
+ echo
+ exit 1;
+fi
+
+IFS=',' read -ra flink_version_list <<< "${flink_versions}"
+IFS=',' read -ra mode_list <<< "${modes}"
+IFS=',' read -ra namespace_list <<< "${namespaces}"
+
+for flink_version in ${flink_version_list[@]}; do
+ test_image=${image}
+ if [[ -z ${test_image} ]]; then
+ position=$(get_position ${flink_version} ${versions[@]})
+ if [[ -z ${position} ]]; then
+ echo
+ echo "-----------------------------------"
+ echo "Unknown Flink version ${flink_version}, please provide an image or a known Flink version"
+ echo "-----------------------------------"
+ echo
+ exit 1;
+ fi
+ test_image=${images[${position}]};
+ fi
+
+ for mode in ${mode_list[@]}; do
+ prepare_tests ${test_image} ${flink_version} ${mode}
+ for namespace in ${namespace_list[@]}; do
+ for script in ${scripts}; do
+ if [[ ${script} = "test_multi_sessionjob.sh" && ${namespace} = "default" ]]; then
Review Comment:
Please extend the message or at least add a comment why the test is skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gaborgsomogyi commented on pull request #455: FLINK-30199: Add a script to run Kubernetes Operator e2e tests manually
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #455:
URL: https://github.com/apache/flink-kubernetes-operator/pull/455#issuecomment-1332097122
This change is a great help, thanks guys! 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] mbalassi merged pull request #455: FLINK-30199: Add a script to run Kubernetes Operator e2e tests manually
Posted by GitBox <gi...@apache.org>.
mbalassi merged PR #455:
URL: https://github.com/apache/flink-kubernetes-operator/pull/455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org