[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1223: HDDS-3990. Test Kubernetes examples with acceptance tests

[GitBox <gi...@apache.org>]

adoroszlai commented on a change in pull request #1223:
URL: https://github.com/apache/hadoop-ozone/pull/1223#discussion_r460662062



##########
File path: hadoop-ozone/dist/src/main/k8s/examples/getting-started/test.sh
##########
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export K8S_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+cd "$K8S_DIR"
+
+# shellcheck source=/dev/null
+source "../testlib.sh"
+
+rm -rf result
+
+regenerate_resources
+
+start_k8s_env
+
+execute_robot_test scm-0 smoketest/basic/basic.robot
+
+combine_reports 'getting-started'
+
+stop_k8s_env
+
+flekszible generate

Review comment:
       If I understand correctly, this `flekszible generate` call is only needed when executed from source dir (`hadoop-ozone/dist/src/main/k8s`) to restore source files.  I propose it to be executed as part of `stop_k8s_env`:
   
   1. I ran tests from `target/...`, had some errors and tried to find out what's wrong.  It was confusing to see resource files referencing non-existent docker image `apache/ozone:0.6.0-SNAPSHOT` (plus other differences compared to the files actually used for the test).
   2. Avoid possible omission in new scripts.
   3. Reduce code duplication.
   4. Save some very minimal runtime cost.

##########
File path: hadoop-ozone/dist/src/main/k8s/examples/testlib.sh
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+retry() {
+   n=0
+   until [ $n -ge 30 ]
+   do
+      "$@" && break
+      n=$[$n+1]
+      echo "$n '$@' is failed..."
+      sleep ${RETRY_SLEEP:-3}
+   done
+   if [ $n -eq 30 ]; then
+      return 255
+   fi
+}
+
+grep_log() {
+   CONTAINER="$1"
+   PATTERN="$2"
+   kubectl logs "$1"  | grep "$PATTERN"
+}
+
+wait_for_startup(){
+   print_phase "Waiting until the k8s cluster is running"
+   retry all_pods_are_running
+   retry grep_log scm-0 "SCM exiting safe mode."
+   retry grep_log om-0 "HTTP server of ozoneManager listening"
+   print_phase "Cluster is up and running"
+}
+
+all_pods_are_running() {
+   RUNNING_COUNT=$(kubectl get pod --field-selector status.phase=Running | wc -l)
+   ALL_COUNT=$(kubectl get pod | wc -l)
+   RUNNING_COUNT=$((RUNNING_COUNT - 1))
+   ALL_COUNT=$((ALL_COUNT - 1))
+   if [ "$RUNNING_COUNT" -lt "3" ]; then
+      echo "$RUNNING_COUNT pods are running. Waiting for more."
+      return 1
+   elif [ "$RUNNING_COUNT" -ne "$ALL_COUNT" ]; then
+      echo "$RUNNING_COUNT pods are running out from the $ALL_COUNT"
+      return 2
+   else
+      STARTED=true
+      return 0
+   fi
+}
+
+start_k8s_env() {
+   print_phase "Deleting existing k8s resources"
+   #reset environment
+   kubectl delete pvc --all
+   kubectl delete pv --all
+   kubectl delete statefulset --all
+   kubectl delete daemonset --all
+   kubectl delete deployment --all
+   kubectl delete service --all
+   kubectl delete configmap --all
+   kubectl delete pod --all
+
+   print_phase "Applying k8s resources from $1"
+   kubectl apply -f .
+   wait_for_startup
+}
+
+stop_k8s_env() {
+   if [ ! "$KEEP_RUNNING" ]; then
+     kubectl delete -f .
+   fi
+}
+
+regenerate_resources() {
+  PARENT_OF_PARENT=$(realpath ../..)
+
+  if [ $(basename $PARENT_OF_PARENT) == "k8s" ]; then
+    #running from src dir
+    OZONE_ROOT=$(realpath ../../../../../target/ozone-0.6.0-SNAPSHOT)
+  else
+    #running from dist
+    OZONE_ROOT=$(realpath ../../..)
+  fi
+
+  flekszible generate -t mount:hostPath="$OZONE_ROOT",path=/opt/hadoop -t image:image=apache/ozone-runner:20200420-1 -t ozone/onenode

Review comment:
       Is there any way to avoid hard-coding the image name?

##########
File path: hadoop-ozone/dist/src/main/k8s/examples/testlib.bats
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+
+#
+# Can be executed with bats (https://github.com/bats-core/bats-core)
+# bats gc_opts.bats (FROM THE CURRENT DIRECTORY)

Review comment:
       ```suggestion
   # bats testlib.bats (FROM THE CURRENT DIRECTORY)
   ```

##########
File path: hadoop-ozone/dist/src/main/k8s/examples/getting-started/test.sh
##########
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export K8S_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+cd "$K8S_DIR"
+
+# shellcheck source=/dev/null
+source "../testlib.sh"
+
+rm -rf result
+
+regenerate_resources
+
+start_k8s_env
+
+execute_robot_test scm-0 smoketest/basic/basic.robot
+
+combine_reports 'getting-started'

Review comment:
       Argument is not needed since `combine_reports` uses `pwd` instead of parameter -- applies to all `test.sh` scripts.

##########
File path: hadoop-ozone/dist/src/main/k8s/examples/testlib.sh
##########
@@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+retry() {
+   n=0
+   until [ $n -ge 30 ]
+   do
+      "$@" && break
+      n=$[$n+1]
+      echo "$n '$@' is failed..."
+      sleep ${RETRY_SLEEP:-3}
+   done
+   if [ $n -eq 30 ]; then
+      return 255
+   fi
+}
+
+grep_log() {
+   CONTAINER="$1"
+   PATTERN="$2"
+   kubectl logs "$1"  | grep "$PATTERN"
+}
+
+wait_for_startup(){
+   print_phase "Waiting until the k8s cluster is running"
+   retry all_pods_are_running
+   retry grep_log scm-0 "SCM exiting safe mode."
+   retry grep_log om-0 "HTTP server of ozoneManager listening"
+   print_phase "Cluster is up and running"
+}
+
+all_pods_are_running() {
+   RUNNING_COUNT=$(kubectl get pod --field-selector status.phase=Running | wc -l)
+   ALL_COUNT=$(kubectl get pod | wc -l)
+   RUNNING_COUNT=$((RUNNING_COUNT - 1))
+   ALL_COUNT=$((ALL_COUNT - 1))
+   if [ "$RUNNING_COUNT" -lt "3" ]; then
+      echo "$RUNNING_COUNT pods are running. Waiting for more."
+      return 1
+   elif [ "$RUNNING_COUNT" -ne "$ALL_COUNT" ]; then
+      echo "$RUNNING_COUNT pods are running out from the $ALL_COUNT"
+      return 2
+   else
+      STARTED=true
+      return 0
+   fi
+}
+
+start_k8s_env() {
+   print_phase "Deleting existing k8s resources"
+   #reset environment
+   kubectl delete pvc --all
+   kubectl delete pv --all
+   kubectl delete statefulset --all
+   kubectl delete daemonset --all
+   kubectl delete deployment --all
+   kubectl delete service --all
+   kubectl delete configmap --all
+   kubectl delete pod --all
+
+   print_phase "Applying k8s resources from $1"
+   kubectl apply -f .
+   wait_for_startup
+}
+
+stop_k8s_env() {
+   if [ ! "$KEEP_RUNNING" ]; then
+     kubectl delete -f .
+   fi
+}
+
+regenerate_resources() {
+  PARENT_OF_PARENT=$(realpath ../..)
+
+  if [ $(basename $PARENT_OF_PARENT) == "k8s" ]; then
+    #running from src dir
+    OZONE_ROOT=$(realpath ../../../../../target/ozone-0.6.0-SNAPSHOT)
+  else
+    #running from dist
+    OZONE_ROOT=$(realpath ../../..)
+  fi
+
+  flekszible generate -t mount:hostPath="$OZONE_ROOT",path=/opt/hadoop -t image:image=apache/ozone-runner:20200420-1 -t ozone/onenode
+}
+
+execute_robot_test() {
+   print_phase "Executing robot tests $@"
+   mkdir -p result
+
+   CONTAINER="$1"
+   shift 1 #Remove first argument which was the container name
+
+   # shellcheck disable=SC2206
+   ARGUMENTS=($@)
+
+   kubectl exec -it "${CONTAINER}" -- bash -c 'rm -rf /tmp/report'
+   kubectl exec -it "${CONTAINER}" -- bash -c 'mkdir -p  /tmp/report'
+   kubectl exec -it "${CONTAINER}" -- robot --nostatusrc -d /tmp/report ${ARGUMENTS[@]} || true
+   kubectl cp "${CONTAINER}":/tmp/report/output.xml "result/$CONTAINER-$RANDOM.xml" || true

Review comment:
       `$RANDOM`: TIL, thanks. 👍 

##########
File path: .github/workflows/post-commit.yml
##########
@@ -224,3 +224,60 @@ jobs:
          with:
            name: coverage
            path: target/coverage
+  kubernetes:
+    name: kubernetes
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/cache@v2
+        with:
+          path: ~/.m2/repository
+          key: maven-repo-${{ hashFiles('**/pom.xml') }}
+      - uses: actions/cache@v2
+        with:
+          path: |
+            ~/.pnpm-store
+            **/node_modules
+          key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
+          restore-keys: |
+            ${{ runner.os }}-pnpm-
+      - name: checkout to /mnt/ozone
+        run: |
+          sudo chmod 777 /mnt
+          git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
+          cd /mnt/ozone
+          git fetch origin "${GITHUB_REF}"
+          git checkout FETCH_HEAD
+          git reset --hard
+      - name: insstall robotframework

Review comment:
       Nit: 
   ```suggestion
         - name: install robotframework
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org