You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tr...@apache.org on 2019/12/21 18:20:44 UTC

[flink] branch release-1.10 updated (7d8e9ea -> 4be12d1)

This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a change to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git.


    from 7d8e9ea  [FLINK-10938][e2e] Add e2e test for natively running flink session cluster on kubernetes
     new ffbb9a6  [hotfix] Enable killing mesos job manager in e2e clean up function
     new cbaf8cc  [hotfix] Forcelly remove the useless log in e2e clean up function
     new 43ba73c  [Flink-15135][e2e][Mesos] Adding Dockerfiles relevant to Mesos setup
     new 6959750  [Flink-15135][e2e][Mesos] Adding common functions to setup Flink on Mesos cluster
     new 66720d0  [Flink-15135][e2e][Mesos] Adding WordCount e2e test for Flink on Mesos
     new dad2e26  [hotfix] Enable arg list for nightly.sh
     new d1adee2  [Flink-15135][e2e][Mesos] Enable Mesos e2e test in Travis
     new 5a2b6fc  [hotfix] Adding Mesos e2e tests to run-nightly-test script
     new 4be12d1  [hotfix] Using retry_times when building docker image

The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .travis.yml                                        |  2 +-
 flink-end-to-end-tests/run-nightly-tests.sh        |  6 ++
 flink-end-to-end-tests/test-scripts/common.sh      | 29 +++++----
 ...st_streaming_gcp_pubsub.sh => common_docker.sh} | 16 ++++-
 .../test-scripts/common_mesos_docker.sh            | 72 ++++++++++++++++++++++
 .../test-scripts/common_yarn_docker.sh             | 25 +++-----
 .../krb5.conf => docker-mesos-cluster/Dockerfile}  | 34 +++++-----
 .../docker-mesos-cluster/docker-compose.yml        | 69 +++++++++++++++++++++
 .../test-scripts/test-runner-common.sh             |  2 +-
 ...t_batch_allround.sh => test_mesos_wordcount.sh} | 22 +++----
 tools/travis/nightly.sh                            |  3 +-
 tools/travis/splits/split_container.sh             |  4 ++
 12 files changed, 220 insertions(+), 64 deletions(-)
 copy flink-end-to-end-tests/test-scripts/{test_streaming_gcp_pubsub.sh => common_docker.sh} (74%)
 mode change 100755 => 100644
 create mode 100644 flink-end-to-end-tests/test-scripts/common_mesos_docker.sh
 copy flink-end-to-end-tests/test-scripts/{docker-hadoop-secure-cluster/config/krb5.conf => docker-mesos-cluster/Dockerfile} (54%)
 create mode 100644 flink-end-to-end-tests/test-scripts/docker-mesos-cluster/docker-compose.yml
 copy flink-end-to-end-tests/test-scripts/{test_batch_allround.sh => test_mesos_wordcount.sh} (62%)


[flink] 05/09: [Flink-15135][e2e][Mesos] Adding WordCount e2e test for Flink on Mesos

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 66720d042a80dddd8c2c754cdd9c7bf41e38dceb
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Thu Dec 12 11:08:24 2019 +0800

    [Flink-15135][e2e][Mesos] Adding WordCount e2e test for Flink on Mesos
---
 .../test-scripts/test_mesos_wordcount.sh           | 36 ++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/flink-end-to-end-tests/test-scripts/test_mesos_wordcount.sh b/flink-end-to-end-tests/test-scripts/test_mesos_wordcount.sh
new file mode 100755
index 0000000..d2f5c40
--- /dev/null
+++ b/flink-end-to-end-tests/test-scripts/test_mesos_wordcount.sh
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+set -o pipefail
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_mesos_docker.sh
+
+RESULT_HASH="72a690412be8928ba239c2da967328a5"
+INPUT_ARGS="--input ${TEST_INFRA_DIR}/test-data/words"
+OUTPUT_LOCATION="${TEST_DATA_DIR}/out/wc_out_mesos"
+TEST_PROGRAM_JAR=${FLINK_DIR}/examples/batch/WordCount.jar
+
+mkdir -p "${TEST_DATA_DIR}"
+
+start_flink_cluster_with_mesos
+
+docker exec -it mesos-master nohup bash -c "${FLINK_DIR}/bin/flink run -p 1 ${TEST_PROGRAM_JAR} ${INPUT_ARGS} --output ${OUTPUT_LOCATION}"
+
+check_result_hash "Mesos WordCount test" "${OUTPUT_LOCATION}" "${RESULT_HASH}"


[flink] 04/09: [Flink-15135][e2e][Mesos] Adding common functions to setup Flink on Mesos cluster

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 6959750a9b81ce2f9757b3e3039c60f2e20ac92e
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Thu Dec 12 11:05:00 2019 +0800

    [Flink-15135][e2e][Mesos] Adding common functions to setup Flink on Mesos cluster
---
 flink-end-to-end-tests/test-scripts/common.sh      | 27 +++++---
 .../test-scripts/common_docker.sh                  | 34 ++++++++++
 .../test-scripts/common_mesos_docker.sh            | 72 ++++++++++++++++++++++
 .../test-scripts/common_yarn_docker.sh             | 11 +---
 4 files changed, 125 insertions(+), 19 deletions(-)

diff --git a/flink-end-to-end-tests/test-scripts/common.sh b/flink-end-to-end-tests/test-scripts/common.sh
index 7ac2f3a..e519056 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -230,27 +230,34 @@ function start_local_zk {
     done < <(grep "^server\." "${FLINK_DIR}/conf/zoo.cfg")
 }
 
-function wait_dispatcher_running {
-  # wait at most 10 seconds until the dispatcher is up
-  local QUERY_URL="${REST_PROTOCOL}://${NODENAME}:8081/taskmanagers"
+function wait_rest_endpoint_up {
+  local query_url=$1
+  local endpoint_name=$2
+  local successful_response_regex=$3
+  # wait at most 10 seconds until the endpoint is up
   local TIMEOUT=20
   for i in $(seq 1 ${TIMEOUT}); do
-    # without the || true this would exit our script if the JobManager is not yet up
-    QUERY_RESULT=$(curl ${CURL_SSL_ARGS} "$QUERY_URL" 2> /dev/null || true)
+    # without the || true this would exit our script if the endpoint is not yet up
+    QUERY_RESULT=$(curl ${CURL_SSL_ARGS} "$query_url" 2> /dev/null || true)
 
-    # ensure the taskmanagers field is there at all and is not empty
-    if [[ ${QUERY_RESULT} =~ \{\"taskmanagers\":\[.+\]\} ]]; then
-      echo "Dispatcher REST endpoint is up."
+    # ensure the response adapts with the suceessful regex
+    if [[ ${QUERY_RESULT} =~ ${successful_response_regex} ]]; then
+      echo "${endpoint_name} REST endpoint is up."
       return
     fi
 
-    echo "Waiting for dispatcher REST endpoint to come up..."
+    echo "Waiting for ${endpoint_name} REST endpoint to come up..."
     sleep 1
   done
-  echo "Dispatcher REST endpoint has not started within a timeout of ${TIMEOUT} sec"
+  echo "${endpoint_name} REST endpoint has not started within a timeout of ${TIMEOUT} sec"
   exit 1
 }
 
+function wait_dispatcher_running {
+  local query_url="${REST_PROTOCOL}://${NODENAME}:8081/taskmanagers"
+  wait_rest_endpoint_up "${query_url}" "Dispatcher" "\{\"taskmanagers\":\[.+\]\}"
+}
+
 function start_cluster {
   "$FLINK_DIR"/bin/start-cluster.sh
   wait_dispatcher_running
diff --git a/flink-end-to-end-tests/test-scripts/common_docker.sh b/flink-end-to-end-tests/test-scripts/common_docker.sh
new file mode 100644
index 0000000..cf3a798
--- /dev/null
+++ b/flink-end-to-end-tests/test-scripts/common_docker.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+set -o pipefail
+
+source "$(dirname "$0")"/common.sh
+
+docker --version
+docker-compose --version
+
+function containers_health_check() {
+  local container_names=${@:1}
+  for container in ${container_names}; do
+    if ! [ $(docker inspect -f '{{.State.Running}}' ${container} 2>&1) = 'true' ];
+    then
+      return 1;
+    fi
+  done
+}
diff --git a/flink-end-to-end-tests/test-scripts/common_mesos_docker.sh b/flink-end-to-end-tests/test-scripts/common_mesos_docker.sh
new file mode 100644
index 0000000..a2f9352
--- /dev/null
+++ b/flink-end-to-end-tests/test-scripts/common_mesos_docker.sh
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+set -o pipefail
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_docker.sh
+
+MAX_RETRY_SECONDS=120
+IMAGE_BUILD_RETRIES=5
+NODENAME=${NODENAME:-`hostname -f`}
+
+export INPUT_VOLUME=${END_TO_END_DIR}/test-scripts/test-data
+
+echo "End-to-end directory $END_TO_END_DIR"
+
+start_time=$(date +%s)
+
+# make sure we stop our cluster at the end
+function cluster_shutdown {
+  docker-compose -f $END_TO_END_DIR/test-scripts/docker-mesos-cluster/docker-compose.yml down
+  rm -rf ${INPUT_VOLUME}/log
+  rm -rf ${INPUT_VOLUME}/tmp
+}
+on_exit cluster_shutdown
+
+function start_flink_cluster_with_mesos() {
+    echo "Starting Flink on Mesos cluster"
+    build_image
+
+    docker-compose -f $END_TO_END_DIR/test-scripts/docker-mesos-cluster/docker-compose.yml up -d
+
+    # wait for the Mesos master and slave set up
+    start_time=$(date +%s)
+    wait_rest_endpoint_up "http://${NODENAME}:5050/slaves" "Mesos" "\{\"slaves\":\[.+\].*\}"
+
+    # perform health checks
+    containers_health_check "mesos-master" "mesos-slave"
+
+    set_config_key "jobmanager.rpc.address" "mesos-master"
+    set_config_key "rest.address" "mesos-master"
+
+    docker exec -it mesos-master nohup bash -c "${FLINK_DIR}/bin/mesos-appmaster.sh -Dmesos.master=mesos-master:5050 &"
+    wait_rest_endpoint_up "http://${NODENAME}:8081/taskmanagers" "Dispatcher" "\{\"taskmanagers\":\[.*\]\}"
+    return 0
+}
+
+function build_image() {
+    echo "Building Mesos Docker container"
+    if ! retry_times $IMAGE_BUILD_RETRIES 0 docker build -f $END_TO_END_DIR/test-scripts/docker-mesos-cluster/Dockerfile \
+        -t flink/docker-mesos-cluster:latest \
+        $END_TO_END_DIR/test-scripts/docker-mesos-cluster/; then
+        echo "ERROR: Could not build mesos image. Aborting..."
+        exit 1
+    fi
+}
diff --git a/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh b/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
index 1cfb4a7..3e5f15a 100755
--- a/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
+++ b/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
@@ -19,6 +19,7 @@
 set -o pipefail
 
 source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_docker.sh
 
 FLINK_TARBALL_DIR=$TEST_DATA_DIR
 FLINK_TARBALL=flink.tar.gz
@@ -31,8 +32,6 @@ echo "Flink Tarball directory $FLINK_TARBALL_DIR"
 echo "Flink tarball filename $FLINK_TARBALL"
 echo "Flink distribution directory name $FLINK_DIRNAME"
 echo "End-to-end directory $END_TO_END_DIR"
-docker --version
-docker-compose --version
 
 start_time=$(date +%s)
 
@@ -62,13 +61,7 @@ function start_hadoop_cluster() {
     done
 
     # perform health checks
-    if ! { [ $(docker inspect -f '{{.State.Running}}' master 2>&1) = 'true' ] &&
-           [ $(docker inspect -f '{{.State.Running}}' slave1 2>&1) = 'true' ] &&
-           [ $(docker inspect -f '{{.State.Running}}' slave2 2>&1) = 'true' ] &&
-           [ $(docker inspect -f '{{.State.Running}}' kdc 2>&1) = 'true' ]; };
-    then
-        return 1
-    fi
+    containers_health_check "master" "slave1" "slave2" "kdc"
 
     # try and see if NodeManagers are up, otherwise the Flink job will not have enough resources
     # to run


[flink] 03/09: [Flink-15135][e2e][Mesos] Adding Dockerfiles relevant to Mesos setup

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 43ba73cd74b66903c8683723ba184379c7f910e3
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Thu Dec 12 10:58:58 2019 +0800

    [Flink-15135][e2e][Mesos] Adding Dockerfiles relevant to Mesos setup
---
 .../test-scripts/docker-mesos-cluster/Dockerfile   | 36 +++++++++++
 .../docker-mesos-cluster/docker-compose.yml        | 69 ++++++++++++++++++++++
 2 files changed, 105 insertions(+)

diff --git a/flink-end-to-end-tests/test-scripts/docker-mesos-cluster/Dockerfile b/flink-end-to-end-tests/test-scripts/docker-mesos-cluster/Dockerfile
new file mode 100644
index 0000000..ea36295
--- /dev/null
+++ b/flink-end-to-end-tests/test-scripts/docker-mesos-cluster/Dockerfile
@@ -0,0 +1,36 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# Creates Mesos cluster on Docker
+
+FROM ubuntu:xenial
+
+RUN apt-get update && \
+  apt-get -y install apt-transport-https ca-certificates && \
+  apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
+  echo deb https://apt.dockerproject.org/repo ubuntu-xenial main > /etc/apt/sources.list.d/docker.list && \
+  apt-get update && \
+  apt-get -y install curl docker-engine
+
+RUN echo "deb http://repos.mesosphere.io/ubuntu/ xenial main" > /etc/apt/sources.list.d/mesosphere.list && \
+  apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \
+  apt-get -y update && \
+  apt-get -y install openjdk-8-jdk && \
+  apt-get -y install mesos=1.7.1-2.0.1 openjdk-9-jre-headless- && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
diff --git a/flink-end-to-end-tests/test-scripts/docker-mesos-cluster/docker-compose.yml b/flink-end-to-end-tests/test-scripts/docker-mesos-cluster/docker-compose.yml
new file mode 100644
index 0000000..cab5f56
--- /dev/null
+++ b/flink-end-to-end-tests/test-scripts/docker-mesos-cluster/docker-compose.yml
@@ -0,0 +1,69 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+version: '3.5'
+
+networks:
+  docker-mesos-cluster-network:
+    name: docker-mesos-cluster-network
+
+services:
+  master:
+    image: flink/docker-mesos-cluster:latest
+    entrypoint: mesos-master
+    container_name: "mesos-master"
+    networks:
+      docker-mesos-cluster-network:
+        aliases:
+          - mesos-master
+    ports:
+      - "8081:8081"
+      - "5050:5050"
+    volumes:
+      - ${INPUT_VOLUME}/log/mesos:/var/log/mesos
+      - ${INPUT_VOLUME}/tmp/mesos:/var/tmp/mesos
+      - ${END_TO_END_DIR}:${END_TO_END_DIR}
+      - ${FLINK_DIR}:${FLINK_DIR}
+    environment:
+      MESOS_PORT: 5050
+      MESOS_QUORUM: 1
+      MESOS_REGISTRY: in_memory
+      MESOS_LOG_DIR: /var/log/mesos
+      MESOS_WORK_DIR: /var/tmp/mesos
+
+  slave:
+    image: flink/docker-mesos-cluster:latest
+    entrypoint: mesos-slave
+    privileged: true
+    depends_on:
+      - master
+    container_name: "mesos-slave"
+    networks:
+      - docker-mesos-cluster-network
+    volumes:
+      - ${INPUT_VOLUME}/log/mesos-sl:/var/log/mesos
+      - ${INPUT_VOLUME}/tmp/mesos-sl:/var/tmp/mesos
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ${END_TO_END_DIR}:${END_TO_END_DIR}
+    environment:
+      MESOS_PORT: 5051
+      MESOS_MASTER: mesos-master:5050
+      MESOS_SWITCH_USER: 0
+      MESOS_CONTAINERIZERS: mesos
+      MESOS_LOG_DIR: /var/log/mesos
+      MESOS_WORK_DIR: /var/tmp/mesos
+      MESOS_SYSTEMD_ENABLE_SUPPORT: 'false'


[flink] 08/09: [hotfix] Adding Mesos e2e tests to run-nightly-test script

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 5a2b6fc292bcf1bf8a2a3f30bdb57565d4bc7b9d
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Fri Dec 13 10:09:04 2019 +0800

    [hotfix] Adding Mesos e2e tests to run-nightly-test script
---
 flink-end-to-end-tests/run-nightly-tests.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/flink-end-to-end-tests/run-nightly-tests.sh b/flink-end-to-end-tests/run-nightly-tests.sh
index 7c8d181..4adb640 100755
--- a/flink-end-to-end-tests/run-nightly-tests.sh
+++ b/flink-end-to-end-tests/run-nightly-tests.sh
@@ -115,6 +115,12 @@ run_test "Run Kubernetes test" "$END_TO_END_DIR/test-scripts/test_kubernetes_emb
 run_test "Run kubernetes session test" "$END_TO_END_DIR/test-scripts/test_kubernetes_session.sh"
 
 ################################################################################
+# Mesos
+################################################################################
+
+run_test "Run Mesos WordCount test" "$END_TO_END_DIR/test-scripts/test_mesos_wordcount.sh"
+
+################################################################################
 # Miscellaneous
 ################################################################################
 


[flink] 06/09: [hotfix] Enable arg list for nightly.sh

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit dad2e262365696cf6c95cde353aacb7cef074439
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Sat Dec 21 12:40:59 2019 +0800

    [hotfix] Enable arg list for nightly.sh
---
 tools/travis/nightly.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/travis/nightly.sh b/tools/travis/nightly.sh
index 0fb918e..4b7fc4b 100755
--- a/tools/travis/nightly.sh
+++ b/tools/travis/nightly.sh
@@ -26,6 +26,7 @@ if [ -z "${HERE}" ] ; then
 fi
 
 SCRIPT=$1
+CMD=${@:2}
 
 source ${HERE}/setup_docker.sh
 source ${HERE}/setup_kubernetes.sh
@@ -71,7 +72,7 @@ if [ $EXIT_CODE == 0 ]; then
 	printf "Running end-to-end tests\n"
 	printf "==============================================================================\n"
 
-	FLINK_DIR=build-target flink-end-to-end-tests/${SCRIPT}
+	FLINK_DIR=build-target flink-end-to-end-tests/${SCRIPT} ${CMD}
 
 	EXIT_CODE=$?
 else


[flink] 02/09: [hotfix] Forcelly remove the useless log in e2e clean up function

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit cbaf8cc449d7a99e44e49cf227456a70d7534d65
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Thu Dec 12 11:11:29 2019 +0800

    [hotfix] Forcelly remove the useless log in e2e clean up function
---
 flink-end-to-end-tests/test-scripts/test-runner-common.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/flink-end-to-end-tests/test-scripts/test-runner-common.sh b/flink-end-to-end-tests/test-scripts/test-runner-common.sh
index 139d2cc..0c2042c 100644
--- a/flink-end-to-end-tests/test-scripts/test-runner-common.sh
+++ b/flink-end-to-end-tests/test-scripts/test-runner-common.sh
@@ -101,7 +101,7 @@ function cleanup_proc {
 
 # Cleans up all temporary folders and files
 function cleanup_tmp_files {
-    rm ${FLINK_DIR}/log/*
+    rm -f ${FLINK_DIR}/log/*
     echo "Deleted all files under ${FLINK_DIR}/log/"
 
     rm -rf ${TEST_DATA_DIR} 2> /dev/null


[flink] 07/09: [Flink-15135][e2e][Mesos] Enable Mesos e2e test in Travis

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit d1adee21d74e18cfe4d01f23ed16b8ea2591d405
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Thu Dec 12 11:16:58 2019 +0800

    [Flink-15135][e2e][Mesos] Enable Mesos e2e test in Travis
    
    This closes #10538.
---
 .travis.yml                            | 2 +-
 tools/travis/splits/split_container.sh | 4 ++++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 34129ab..2ee6e07 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -351,7 +351,7 @@ jobs:
       name: e2e - checkpoints
     - if: type = cron
       env: PROFILE="-DincludeE2E=org.apache.flink.tests.util.categories.TravisGroup5"
-      script: ./tools/travis/nightly.sh split_container.sh
+      script: ./tools/travis/nightly.sh split_container.sh without-hadoop
       name: e2e - container
     - if: type = cron
       env: PROFILE="-DincludeE2E=org.apache.flink.tests.util.categories.TravisGroup6"
diff --git a/tools/travis/splits/split_container.sh b/tools/travis/splits/split_container.sh
index 6ee4368..8ec1a35 100755
--- a/tools/travis/splits/split_container.sh
+++ b/tools/travis/splits/split_container.sh
@@ -24,6 +24,7 @@ if [ -z "$END_TO_END_DIR" ] ; then
     # to the script (e.g. permissions re-evaled after suid)
     exit 1  # fail
 fi
+HADOOP_INTEGRATION=${1-with-hadoop}
 
 export END_TO_END_DIR
 
@@ -48,6 +49,9 @@ run_test "Running Kerberized YARN on Docker test (default input)" "$END_TO_END_D
 run_test "Running Kerberized YARN on Docker test (custom fs plugin)" "$END_TO_END_DIR/test-scripts/test_yarn_kerberos_docker.sh dummy-fs"
 run_test "Run kubernetes test" "$END_TO_END_DIR/test-scripts/test_kubernetes_embedded_job.sh"
 run_test "Run kubernetes session test" "$END_TO_END_DIR/test-scripts/test_kubernetes_session.sh"
+if [[ "${HADOOP_INTEGRATION}" = "with-hadoop" ]]; then
+    run_test "Run Mesos WordCount test" "$END_TO_END_DIR/test-scripts/test_mesos_wordcount.sh"
+fi
 
 printf "\n[PASS] All tests passed\n"
 exit 0


[flink] 09/09: [hotfix] Using retry_times when building docker image

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 4be12d1609a1ec0c470000b970f8694d9cdf7f6a
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Sun Dec 15 09:04:59 2019 +0800

    [hotfix] Using retry_times when building docker image
---
 flink-end-to-end-tests/test-scripts/common_yarn_docker.sh | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh b/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
index 3e5f15a..dcd28cc 100755
--- a/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
+++ b/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
@@ -27,6 +27,7 @@ FLINK_DIRNAME=$(basename $FLINK_DIR)
 
 MAX_RETRY_SECONDS=120
 CLUSTER_SETUP_RETRIES=3
+IMAGE_BUILD_RETRIES=5
 
 echo "Flink Tarball directory $FLINK_TARBALL_DIR"
 echo "Flink tarball filename $FLINK_TARBALL"
@@ -90,16 +91,13 @@ function start_hadoop_cluster() {
 
 function build_image() {
     echo "Building Hadoop Docker container"
-    until docker build --build-arg HADOOP_VERSION=2.8.4 \
+    if ! retry_times $IMAGE_BUILD_RETRIES 2 docker build --build-arg HADOOP_VERSION=2.8.4 \
         -f $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/Dockerfile \
         -t flink/docker-hadoop-secure-cluster:latest \
-        $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/;
-    do
-        # with all the downloading and ubuntu updating a lot of flakiness can happen, make sure
-        # we don't immediately fail
-        echo "Something went wrong while building the Docker image, retrying ..."
-        sleep 2
-    done
+        $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/; then
+        echo "ERROR: Could not build hadoop image. Aborting..."
+        exit 1
+    fi
 }
 
 function start_hadoop_cluster_and_prepare_flink() {


[flink] 01/09: [hotfix] Enable killing mesos job manager in e2e clean up function

Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git

commit ffbb9a6fb85e377ad51a259a8112d45116e5bdfe
Author: Yangze Guo <ka...@gmail.com>
AuthorDate: Thu Dec 12 10:49:05 2019 +0800

    [hotfix] Enable killing mesos job manager in e2e clean up function
---
 flink-end-to-end-tests/test-scripts/common.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/flink-end-to-end-tests/test-scripts/common.sh b/flink-end-to-end-tests/test-scripts/common.sh
index 4079a82..7ac2f3a 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -536,7 +536,7 @@ function tm_watchdog {
 
 # Kills all job manager.
 function jm_kill_all {
-  kill_all 'StandaloneSessionClusterEntrypoint'
+  kill_all 'ClusterEntrypoint'
 }
 
 # Kills all task manager.