You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by di...@apache.org on 2021/08/27 14:41:12 UTC

[flink] branch release-1.13 updated (e34f67e -> 2bf0f36)

This is an automated email from the ASF dual-hosted git repository.

dianfu pushed a change to branch release-1.13
in repository https://gitbox.apache.org/repos/asf/flink.git.


    from e34f67e  [FLINK-23998][legal] Include flink-dist in NoticeFileChecker
     new 1a0c85b  Revert "[FLINK-23984][python][tests] Temporary disable the PyFlink end to end tests due to Python 3.7 was removed from debian"
     new 2bf0f36  [FLINK-23984][python][tests] Update test_kubernetes_pyflink_application to use conda to install Python

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 flink-end-to-end-tests/run-nightly-tests.sh                | 14 +++++++-------
 .../test-scripts/test_kubernetes_pyflink_application.sh    | 12 +++++++-----
 2 files changed, 14 insertions(+), 12 deletions(-)

[flink] 01/02: Revert "[FLINK-23984][python][tests] Temporary disable the PyFlink end to end tests due to Python 3.7 was removed from debian"

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

dianfu pushed a commit to branch release-1.13
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 1a0c85b5329fcfe6943cfa890a2930c29f817522
Author: Dian Fu <di...@apache.org>
AuthorDate: Fri Aug 27 16:10:13 2021 +0800

    Revert "[FLINK-23984][python][tests] Temporary disable the PyFlink end to end tests due to Python 3.7 was removed from debian"
    
    This reverts commit f998da0f4c711f179a766f6978b12fe0e3ad7fae
---
 flink-end-to-end-tests/run-nightly-tests.sh | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/flink-end-to-end-tests/run-nightly-tests.sh b/flink-end-to-end-tests/run-nightly-tests.sh
index 0d656f6..1acbd23 100755
--- a/flink-end-to-end-tests/run-nightly-tests.sh
+++ b/flink-end-to-end-tests/run-nightly-tests.sh
@@ -140,7 +140,7 @@ if [[ ${PROFILE} != *"jdk11"* && ${PROFILE} != *"enable-adaptive-scheduler"* ]];
 
 	if [[ `uname -i` != 'aarch64' ]]; then
 		# Skip PyFlink e2e test, because MiniConda and Pyarrow which Pyflink depends doesn't support aarch64 currently.
-		# run_test "Run kubernetes pyflink application test" "$END_TO_END_DIR/test-scripts/test_kubernetes_pyflink_application.sh"
+		run_test "Run kubernetes pyflink application test" "$END_TO_END_DIR/test-scripts/test_kubernetes_pyflink_application.sh"
 		
 		# Hadoop YARN deosn't support aarch64 at this moment. See: https://issues.apache.org/jira/browse/HADOOP-16723
 
@@ -232,13 +232,13 @@ run_test "Dependency shading of table modules test" "$END_TO_END_DIR/test-script
 
 run_test "Shaded Hadoop S3A with credentials provider end-to-end test" "$END_TO_END_DIR/test-scripts/test_batch_wordcount.sh hadoop_with_provider"
 
-#if [[ `uname -i` != 'aarch64' ]]; then
-#    run_test "PyFlink end-to-end test" "$END_TO_END_DIR/test-scripts/test_pyflink.sh" "skip_check_exceptions"
-#fi
+if [[ `uname -i` != 'aarch64' ]]; then
+    run_test "PyFlink end-to-end test" "$END_TO_END_DIR/test-scripts/test_pyflink.sh" "skip_check_exceptions"
+fi
 # These tests are known to fail on JDK11. See FLINK-13719
-#if [[ ${PROFILE} != *"jdk11"* ]] && [[ `uname -i` != 'aarch64' ]]; then
-#    run_test "PyFlink YARN per-job on Docker test" "$END_TO_END_DIR/test-scripts/test_pyflink_yarn.sh" "skip_check_exceptions"
-#fi
+if [[ ${PROFILE} != *"jdk11"* ]] && [[ `uname -i` != 'aarch64' ]]; then
+    run_test "PyFlink YARN per-job on Docker test" "$END_TO_END_DIR/test-scripts/test_pyflink_yarn.sh" "skip_check_exceptions"
+fi
 
 ################################################################################
 # Sticky Scheduling

[flink] 02/02: [FLINK-23984][python][tests] Update test_kubernetes_pyflink_application to use conda to install Python

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

dianfu pushed a commit to branch release-1.13
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 2bf0f369391a08eb0df5aba809cca12da1da98e9
Author: Dian Fu <di...@apache.org>
AuthorDate: Fri Aug 27 16:12:19 2021 +0800

    [FLINK-23984][python][tests] Update test_kubernetes_pyflink_application to use conda to install Python
    
    This closes #17014.
---
 .../test-scripts/test_kubernetes_pyflink_application.sh      | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh b/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh
index 87ff034..f62ed43 100755
--- a/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh
+++ b/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh
@@ -77,17 +77,19 @@ PYFLINK_DOCKER_DIR="$TEST_DATA_DIR/pyflink_docker"
 mkdir -p "$PYFLINK_DOCKER_DIR"
 cp "${FLINK_PYTHON_DIR}/dist/${PYFLINK_PACKAGE_FILE}" $PYFLINK_DOCKER_DIR/
 cp "${FLINK_PYTHON_DIR}/apache-flink-libraries/dist/${PYFLINK_LIBRARIES_PACKAGE_FILE}" $PYFLINK_DOCKER_DIR/
+cp "${FLINK_PYTHON_DIR}/dev/lint-python.sh" $PYFLINK_DOCKER_DIR/
 if [[ -d "dist" ]]; then rm -Rf dist; fi
 cd ${PYFLINK_DOCKER_DIR}
 echo "FROM ${PURE_FLINK_IMAGE_NAME}" >> Dockerfile
-echo "RUN apt-get update -y && apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*" >> Dockerfile
-echo "RUN ln -s /usr/bin/python3 /usr/bin/python" >> Dockerfile
+echo "COPY lint-python.sh /tmp/lint-python.sh" >> Dockerfile
+echo "RUN bash /tmp/lint-python.sh -s basic" >> Dockerfile
 echo "COPY ${PYFLINK_PACKAGE_FILE} ${PYFLINK_PACKAGE_FILE}" >> Dockerfile
 echo "COPY ${PYFLINK_LIBRARIES_PACKAGE_FILE} ${PYFLINK_LIBRARIES_PACKAGE_FILE}" >> Dockerfile
-echo "RUN pip3 install ${PYFLINK_LIBRARIES_PACKAGE_FILE}" >> Dockerfile
-echo "RUN pip3 install ${PYFLINK_PACKAGE_FILE}" >> Dockerfile
-echo "RUN rm ${PYFLINK_PACKAGE_FILE}" >> Dockerfile
+echo "RUN /tmp/.conda/bin/python -m pip install ${PYFLINK_LIBRARIES_PACKAGE_FILE}" >> Dockerfile
+echo "RUN /tmp/.conda/bin/python -m pip install ${PYFLINK_PACKAGE_FILE}" >> Dockerfile
 echo "RUN rm ${PYFLINK_LIBRARIES_PACKAGE_FILE}" >> Dockerfile
+echo "RUN rm ${PYFLINK_PACKAGE_FILE}" >> Dockerfile
+echo "ENV PATH /tmp/.conda/bin:\$PATH" >> Dockerfile
 docker build --no-cache --network="host" -t ${PYFLINK_IMAGE_NAME} .
 
 kubectl create clusterrolebinding ${CLUSTER_ROLE_BINDING} --clusterrole=edit --serviceaccount=default:default --namespace=default