[GitHub] [flink] zentol commented on a change in pull request #12268: [FLINK-17375] Refactor travis_watchdog.sh into separate ci and azure scripts.

[GitBox <gi...@apache.org>]

zentol commented on a change in pull request #12268:
URL: https://github.com/apache/flink/pull/12268#discussion_r429879700



##########
File path: tools/ci/watchdog.sh
##########
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains a watchdog tool to run monitor and potentially kill tasks
+# not producing any output for n seconds.
+#
+
+# Number of seconds w/o output before printing a stack trace and killing the watched process
+MAX_NO_OUTPUT=${MAX_NO_OUTPUT:-900}
+
+# Number of seconds to sleep before checking the output again
+SLEEP_TIME=${SLEEP_TIME:-20}
+
+CMD_OUT=${CMD_OUT:-"/tmp/watchdog.out"}
+CMD_PID=${CMD_PID:-"/tmp/watchdog.pid"}
+CMD_EXIT=${CMD_EXIT:-"/tmp/watchdog.exit"}

Review comment:
       are these actually set anywhere?

##########
File path: tools/ci/test_controller.sh
##########
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains generic control over the test execution.
+#
+
+HERE="`dirname \"$0\"`"             # relative
+HERE="`( cd \"$HERE\" && pwd )`"    # absolutized and normalized
+if [ -z "$HERE" ] ; then
+	exit 1
+fi
+
+source "${HERE}/stage.sh"
+source "${HERE}/maven-utils.sh"
+source "${HERE}/controller_utils.sh"
+source "${HERE}/watchdog.sh"
+TEST=$1
+
+# =============================================================================
+# Step 0: Check & print environment information & configure env
+# =============================================================================
+
+# check preconditions
+if [ -z "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable 'DEBUG_FILES' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+if [ ! -d "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable DEBUG_FILES=$DEBUG_FILES points to a directory that does not exist"
+	exit 1
+fi
+
+if [ -z "$TEST" ] ; then
+	echo "ERROR: Environment variable 'TEST' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+echo "Printing environment information"
+
+echo "PATH=$PATH"
+run_mvn -version
+echo "Commit: $(git rev-parse HEAD)"
+print_system_info
+
+# enable coredumps for this process
+ulimit -c unlimited
+
+# configure JVMs to produce heap dumps
+export JAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError"
+
+# some tests provide additional logs if they find this variable
+export IS_CI=true
+
+# =============================================================================
+# Step 1: Compile Flink (again)
+# =============================================================================
+
+WATCHDOG_CALLBACK_ON_TIMEOUT="print_stacktraces | tee ${DEBUG_FILES}/jps-traces.out"
+
+LOG4J_PROPERTIES=${HERE}/log4j-ci.properties
+MVN_LOGGING_OPTIONS="-Dlog.dir=${DEBUG_FILES} -Dlog4j.configurationFile=file://$LOG4J_PROPERTIES"
+# Maven command to run. We set the forkCount manually, because otherwise Maven sees too many cores
+# on some CI environments. Set forkCountTestPackage to 1 for container-based environment (4 GiB memory)
+# and 2 for sudo-enabled environment (7.5 GiB memory).
+MVN_COMMON_OPTIONS="-Dflink.forkCount=2 -Dflink.forkCountTestPackage=2 -Dfast -Pskip-webui-build $MVN_LOGGING_OPTIONS"
+MVN_COMPILE_OPTIONS="-DskipTests"
+MVN_COMPILE_MODULES=$(get_compile_modules_for_stage ${TEST})
+
+run_with_watchdog "run_mvn $MVN_COMMON_OPTIONS $MVN_COMPILE_OPTIONS $PROFILE $MVN_COMPILE_MODULES install"
+EXIT_CODE=$?
+
+if [ $EXIT_CODE != 0 ]; then
+	echo "=============================================================================="
+	echo "Compilation failure detected, skipping test execution."
+	echo "=============================================================================="
+	exit $EXIT_CODE
+fi
+
+
+# =============================================================================
+# Step 2: Run tests
+# =============================================================================
+
+if [ $TEST == $STAGE_PYTHON ]; then
+	run_with_watchdog "./flink-python/dev/lint-python.sh"
+	EXIT_CODE=$?
+else
+	MVN_TEST_OPTIONS="-Dflink.tests.with-openssl"
+	MVN_TEST_MODULES=$(get_test_modules_for_stage ${TEST})
+
+	run_with_watchdog "run_mvn $MVN_COMMON_OPTIONS $MVN_TEST_OPTIONS $PROFILE $MVN_TEST_MODULES verify"
+	EXIT_CODE=$?
+fi
+
+# =============================================================================
+# Step 3: Put extra logs into $DEBUG_FILES
+# =============================================================================
+
+# only misc builds flink-yarn-tests
+case $TEST in
+	(misc)
+		put_yarn_logs_to_artifacts
+	;;
+esac
+
+collect_coredumps `pwd` $DEBUG_FILES

Review comment:
       nit: Use `$(pwd)` for consistency/clarity?

##########
File path: tools/ci/test_controller.sh
##########
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains generic control over the test execution.
+#
+
+HERE="`dirname \"$0\"`"             # relative
+HERE="`( cd \"$HERE\" && pwd )`"    # absolutized and normalized
+if [ -z "$HERE" ] ; then
+	exit 1
+fi
+
+source "${HERE}/stage.sh"
+source "${HERE}/maven-utils.sh"
+source "${HERE}/controller_utils.sh"
+source "${HERE}/watchdog.sh"
+TEST=$1
+
+# =============================================================================
+# Step 0: Check & print environment information & configure env
+# =============================================================================
+
+# check preconditions
+if [ -z "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable 'DEBUG_FILES' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+if [ ! -d "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable DEBUG_FILES=$DEBUG_FILES points to a directory that does not exist"
+	exit 1
+fi
+
+if [ -z "$TEST" ] ; then
+	echo "ERROR: Environment variable 'TEST' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+echo "Printing environment information"
+
+echo "PATH=$PATH"
+run_mvn -version
+echo "Commit: $(git rev-parse HEAD)"
+print_system_info
+
+# enable coredumps for this process
+ulimit -c unlimited
+
+# configure JVMs to produce heap dumps
+export JAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError"
+
+# some tests provide additional logs if they find this variable
+export IS_CI=true
+
+# =============================================================================
+# Step 1: Compile Flink (again)
+# =============================================================================
+
+WATCHDOG_CALLBACK_ON_TIMEOUT="print_stacktraces | tee ${DEBUG_FILES}/jps-traces.out"

Review comment:
       move this right above `run_with_watchdog`

##########
File path: tools/azure-pipelines/build-python-wheels.yml
##########
@@ -33,14 +32,13 @@ jobs:
         name: set_version
 
       - script: |
-          cd $(Pipeline.Workspace)
-          tar czf $(Pipeline.Workspace)/flink.tar.gz flink_cache/flink-dist/target/flink-$(set_version.VERSION)-bin/flink-$(set_version.VERSION)
+          tar czf $(Pipeline.Workspace)/flink.tar.gz flink-dist/target/flink-$(set_version.VERSION)-bin/flink-$(set_version.VERSION)
         displayName: Compress in tgz
 
       # upload artifacts for building wheels
       - task: PublishPipelineArtifact@1
         inputs:
-          path: $(FLINK_ARTIFACT_DIR)
+          targetPath: $(Pipeline.Workspace)/flink.tar.gz

Review comment:
       why is this set to `targetPath`? jobs-template.yml uses `path`

##########
File path: tools/ci/watchdog.sh
##########
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains a watchdog tool to run monitor and potentially kill tasks
+# not producing any output for n seconds.
+#
+
+# Number of seconds w/o output before printing a stack trace and killing the watched process
+MAX_NO_OUTPUT=${MAX_NO_OUTPUT:-900}
+
+# Number of seconds to sleep before checking the output again
+SLEEP_TIME=${SLEEP_TIME:-20}
+
+CMD_OUT=${CMD_OUT:-"/tmp/watchdog.out"}
+CMD_PID=${CMD_PID:-"/tmp/watchdog.pid"}
+CMD_EXIT=${CMD_EXIT:-"/tmp/watchdog.exit"}
+
+
+# =============================================
+# Utility functions
+# ============================================= 
+
+mod_time () {
+	echo `stat -c "%Y" $CMD_OUT`
+}
+
+the_time() {
+	echo `date +%s`
+}
+
+# watchdog process
+
+watchdog () {
+	touch $CMD_OUT
+
+	while true; do
+		sleep $SLEEP_TIME
+
+		time_diff=$((`the_time` - `mod_time`))
+
+		if [ $time_diff -ge $MAX_NO_OUTPUT ]; then
+			echo "=============================================================================="
+			echo "Process produced no output for ${MAX_NO_OUTPUT} seconds."
+			echo "=============================================================================="
+
+			# run timeout callback
+			$WATCHDOG_CALLBACK_ON_TIMEOUT
+
+			echo "Killing process with pid=$(<$CMD_PID) and all descendants"
+			pkill -P $(<$CMD_PID) # kill descendants
+			kill $(<$CMD_PID) # kill process itself
+
+			exit 1
+		fi
+	done
+}
+
+assume_available () {
+	VAR=$1
+	if [ -z "$VAR" ] ; then
+		echo "ERROR: Environment variable '$VAR' is not set but expected by watchdog.sh"
+		exit 1
+	fi
+}
+
+# =============================================
+# main function
+# =============================================
+
+# entrypoint
+function run_with_watchdog() {
+	local cmd="$1"
+
+	# check preconditions
+	assume_available CMD_OUT # used for writing the process output (to check for activity)
+	assume_available CMD_PID # location of file to write process id to
+	assume_available CMD_EXIT # location of file to writ exit code to
+	assume_available WATCHDOG_CALLBACK_ON_TIMEOUT # bash function to call on timeout
+
+	watchdog &

Review comment:
       I'd prefer explicitly passing the required arguments.

##########
File path: tools/ci/watchdog.sh
##########
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains a watchdog tool to run monitor and potentially kill tasks

Review comment:
       ```suggestion
   # This file contains a watchdog tool to monitor a task and potentially kill it after
   ```

##########
File path: tools/ci/test_controller.sh
##########
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains generic control over the test execution.
+#
+
+HERE="`dirname \"$0\"`"             # relative
+HERE="`( cd \"$HERE\" && pwd )`"    # absolutized and normalized
+if [ -z "$HERE" ] ; then
+	exit 1
+fi
+
+source "${HERE}/stage.sh"
+source "${HERE}/maven-utils.sh"
+source "${HERE}/controller_utils.sh"
+source "${HERE}/watchdog.sh"
+TEST=$1
+
+# =============================================================================
+# Step 0: Check & print environment information & configure env
+# =============================================================================
+
+# check preconditions
+if [ -z "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable 'DEBUG_FILES' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+if [ ! -d "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable DEBUG_FILES=$DEBUG_FILES points to a directory that does not exist"
+	exit 1
+fi
+
+if [ -z "$TEST" ] ; then
+	echo "ERROR: Environment variable 'TEST' is not set but expected by test_controller.sh"

Review comment:
       same as above

##########
File path: tools/ci/stage.sh
##########
@@ -158,6 +158,9 @@ function get_compile_modules_for_stage() {
             # the negation takes precedence, thus not all required modules would be built
             echo ""
         ;;
+        (${STAGE_PYTHON})
+            echo ""

Review comment:
       add a comment for why we compile everything

##########
File path: tools/ci/test_controller.sh
##########
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains generic control over the test execution.
+#
+
+HERE="`dirname \"$0\"`"             # relative
+HERE="`( cd \"$HERE\" && pwd )`"    # absolutized and normalized
+if [ -z "$HERE" ] ; then
+	exit 1
+fi
+
+source "${HERE}/stage.sh"
+source "${HERE}/maven-utils.sh"
+source "${HERE}/controller_utils.sh"
+source "${HERE}/watchdog.sh"
+TEST=$1
+
+# =============================================================================
+# Step 0: Check & print environment information & configure env
+# =============================================================================
+
+# check preconditions
+if [ -z "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable 'DEBUG_FILES' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+if [ ! -d "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable DEBUG_FILES=$DEBUG_FILES points to a directory that does not exist"
+	exit 1
+fi
+
+if [ -z "$TEST" ] ; then
+	echo "ERROR: Environment variable 'TEST' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+echo "Printing environment information"
+
+echo "PATH=$PATH"
+run_mvn -version
+echo "Commit: $(git rev-parse HEAD)"
+print_system_info
+
+# enable coredumps for this process
+ulimit -c unlimited
+
+# configure JVMs to produce heap dumps
+export JAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError"
+
+# some tests provide additional logs if they find this variable
+export IS_CI=true
+
+# =============================================================================
+# Step 1: Compile Flink (again)

Review comment:
       we usually aren't compiling Flink again, but are just rebuilding the jars and installing it into the maven repository.

##########
File path: tools/ci/test_controller.sh
##########
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains generic control over the test execution.
+#
+
+HERE="`dirname \"$0\"`"             # relative
+HERE="`( cd \"$HERE\" && pwd )`"    # absolutized and normalized
+if [ -z "$HERE" ] ; then
+	exit 1
+fi
+
+source "${HERE}/stage.sh"
+source "${HERE}/maven-utils.sh"
+source "${HERE}/controller_utils.sh"
+source "${HERE}/watchdog.sh"
+TEST=$1
+
+# =============================================================================
+# Step 0: Check & print environment information & configure env
+# =============================================================================
+
+# check preconditions
+if [ -z "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable 'DEBUG_FILES' is not set but expected by test_controller.sh"

Review comment:
       add more info on what this is used for; and potentially adjust the name because it is quite non-descriptive

##########
File path: tools/ci/test_controller.sh
##########
@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains generic control over the test execution.
+#
+
+HERE="`dirname \"$0\"`"             # relative
+HERE="`( cd \"$HERE\" && pwd )`"    # absolutized and normalized
+if [ -z "$HERE" ] ; then
+	exit 1
+fi
+
+source "${HERE}/stage.sh"
+source "${HERE}/maven-utils.sh"
+source "${HERE}/controller_utils.sh"
+source "${HERE}/watchdog.sh"
+TEST=$1
+
+# =============================================================================
+# Step 0: Check & print environment information & configure env
+# =============================================================================
+
+# check preconditions
+if [ -z "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable 'DEBUG_FILES' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+if [ ! -d "$DEBUG_FILES" ] ; then
+	echo "ERROR: Environment variable DEBUG_FILES=$DEBUG_FILES points to a directory that does not exist"
+	exit 1
+fi
+
+if [ -z "$TEST" ] ; then
+	echo "ERROR: Environment variable 'TEST' is not set but expected by test_controller.sh"
+	exit 1
+fi
+
+echo "Printing environment information"
+
+echo "PATH=$PATH"
+run_mvn -version
+echo "Commit: $(git rev-parse HEAD)"
+print_system_info
+
+# enable coredumps for this process
+ulimit -c unlimited
+
+# configure JVMs to produce heap dumps
+export JAVA_TOOL_OPTIONS="-XX:+HeapDumpOnOutOfMemoryError"
+
+# some tests provide additional logs if they find this variable
+export IS_CI=true
+
+# =============================================================================
+# Step 1: Compile Flink (again)
+# =============================================================================
+
+WATCHDOG_CALLBACK_ON_TIMEOUT="print_stacktraces | tee ${DEBUG_FILES}/jps-traces.out"
+
+LOG4J_PROPERTIES=${HERE}/log4j-ci.properties
+MVN_LOGGING_OPTIONS="-Dlog.dir=${DEBUG_FILES} -Dlog4j.configurationFile=file://$LOG4J_PROPERTIES"
+# Maven command to run. We set the forkCount manually, because otherwise Maven sees too many cores
+# on some CI environments. Set forkCountTestPackage to 1 for container-based environment (4 GiB memory)

Review comment:
       the information about available memory is likely out-dated.

##########
File path: tools/azure-pipelines/jobs-template.yml
##########
@@ -163,6 +163,7 @@ jobs:
       IT_CASE_S3_SECRET_KEY: $(SECRET_S3_SECRET_KEY)
 
   - task: PublishTestResults@2
+    condition: succeededOrFailed()

Review comment:
       what happens if say, the compilation fails? Is this case handle gracefully?

##########
File path: tools/azure-pipelines/jobs-template.yml
##########
@@ -64,14 +65,20 @@ jobs:
     displayName: "Set to jdk11"
     condition: eq('${{parameters.jdk}}', 'jdk11')
   # Compile
-  - script: STAGE=compile ${{parameters.environment}} ./tools/azure-pipelines/azure_controller.sh compile
-    displayName: Build
+  - script: |
+      ${{parameters.environment}} ./tools/ci/compile.sh
+      EXIT_CODE=$?

Review comment:
       same as below

##########
File path: tools/ci/maven-utils.sh
##########
@@ -73,7 +73,7 @@ function collect_coredumps {
 	echo "Searching for .dump, .dumpstream and related files in '$SEARCHDIR'"
 	for file in `find $SEARCHDIR -type f -regextype posix-extended -iregex '.*\.hprof|.*\.dump|.*\.dumpstream|.*hs.*\.log|.*/core(.[0-9]+)?$'`; do
 		echo "Moving '$file' to target directory ('$TARGET_DIR')"
-		mv $file $TARGET_DIR/
+		mv $file $TARGET_DIR/$(echo $file | tr "/" "-")

Review comment:
       shouldn't be possible to have forward slashes in file names on unix

##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -1068,8 +1067,8 @@ public static void teardown() throws Exception {
 
 	}
 
-	public static boolean isOnTravis() {
-		return System.getenv("TRAVIS") != null && System.getenv("TRAVIS").equals("true");
+	public static boolean isOnCI() {
+		return System.getenv("IS_CI") != null && System.getenv("IS_CI").equals("true");

Review comment:
       hmm, this reminds me, we aren't caching downloads for the e2e tests on azure, are we?

##########
File path: tools/ci/watchdog.sh
##########
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+#
+# This file contains a watchdog tool to run monitor and potentially kill tasks
+# not producing any output for n seconds.
+#
+
+# Number of seconds w/o output before printing a stack trace and killing the watched process
+MAX_NO_OUTPUT=${MAX_NO_OUTPUT:-900}
+
+# Number of seconds to sleep before checking the output again
+SLEEP_TIME=${SLEEP_TIME:-20}
+
+CMD_OUT=${CMD_OUT:-"/tmp/watchdog.out"}
+CMD_PID=${CMD_PID:-"/tmp/watchdog.pid"}
+CMD_EXIT=${CMD_EXIT:-"/tmp/watchdog.exit"}
+
+
+# =============================================
+# Utility functions
+# ============================================= 
+
+mod_time () {
+	echo `stat -c "%Y" $CMD_OUT`
+}
+
+the_time() {
+	echo `date +%s`
+}
+
+# watchdog process
+
+watchdog () {
+	touch $CMD_OUT
+
+	while true; do
+		sleep $SLEEP_TIME
+
+		time_diff=$((`the_time` - `mod_time`))
+
+		if [ $time_diff -ge $MAX_NO_OUTPUT ]; then
+			echo "=============================================================================="
+			echo "Process produced no output for ${MAX_NO_OUTPUT} seconds."
+			echo "=============================================================================="
+
+			# run timeout callback
+			$WATCHDOG_CALLBACK_ON_TIMEOUT
+
+			echo "Killing process with pid=$(<$CMD_PID) and all descendants"
+			pkill -P $(<$CMD_PID) # kill descendants
+			kill $(<$CMD_PID) # kill process itself
+
+			exit 1
+		fi
+	done
+}
+
+assume_available () {
+	VAR=$1
+	if [ -z "$VAR" ] ; then
+		echo "ERROR: Environment variable '$VAR' is not set but expected by watchdog.sh"
+		exit 1
+	fi
+}
+
+# =============================================
+# main function
+# =============================================
+
+# entrypoint
+function run_with_watchdog() {
+	local cmd="$1"
+
+	# check preconditions
+	assume_available CMD_OUT # used for writing the process output (to check for activity)
+	assume_available CMD_PID # location of file to write process id to
+	assume_available CMD_EXIT # location of file to writ exit code to
+	assume_available WATCHDOG_CALLBACK_ON_TIMEOUT # bash function to call on timeout

Review comment:
       if you passed the comment into assume_available you could provide a better error message

##########
File path: tools/azure-pipelines/jobs-template.yml
##########
@@ -121,15 +128,34 @@ jobs:
     continueOnError: true # continue the build even if the cache fails.
     condition: not(eq('${{parameters.test_pool_definition.name}}', 'Default'))
     displayName: Cache Maven local repo
+
   - script: |
       echo "##vso[task.setvariable variable=JAVA_HOME]$JAVA_HOME_11_X64"
       echo "##vso[task.setvariable variable=PATH]$JAVA_HOME_11_X64/bin:$PATH"
     displayName: "Set to jdk11"
     condition: eq('${{parameters.jdk}}', 'jdk11')  
+
   - script: sudo sysctl -w kernel.core_pattern=core.%p
     displayName: Set coredump pattern
+
   # Test
-  - script: STAGE=test ${{parameters.environment}} ./tools/azure-pipelines/azure_controller.sh $(module)
+  - script: |
+      ./tools/azure-pipelines/unpack_build_artifact.sh
+      export DEBUG_FILES="$AGENT_TEMPDIRECTORY/debug_files"

Review comment:
       why isn't all this put into a script?

##########
File path: tools/azure-pipelines/build-python-wheels.yml
##########
@@ -24,7 +24,6 @@ jobs:
       # Compile
       - script: |
           ${{parameters.environment}} ./tools/ci/compile.sh
-          ./tools/azure-pipelines/create_build_artifact.sh

Review comment:
       If python no longer relies on this then a number of exclusions in `create_build_artifact.sh` could probably be removed. Maybe even all of them.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org