You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by kk...@apache.org on 2019/08/01 09:31:34 UTC
[flink] branch master updated (57f7958 -> a33ac4c)
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from 57f7958 [FLINK-12998][docs] Extend debugging_classloading documentation with plugins loading case
new d57ff71 [FLINK-13434][e2e] Change the test_resume_savepoint to use stop-with-savepoint.
new 2369e9c [FLINK-13434][e2e] Change test_stateful_stream_job_upgrade to use stop-with-savepoint.
new b580075 [hotfix][e2e] Harden test_resume_savepoint e2e test.
new a33ac4c [hotfix][e2e] Harden test_stateful_stream_job_upgrade e2e test.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../java/org/apache/flink/client/cli/CliFrontend.java | 6 +++---
flink-end-to-end-tests/test-scripts/common.sh | 4 ++++
.../test-scripts/test_resume_savepoint.sh | 15 +++++++++++++--
.../test-scripts/test_stateful_stream_job_upgrade.sh | 17 ++++++++++++++---
4 files changed, 34 insertions(+), 8 deletions(-)
[flink] 02/04: [FLINK-13434][e2e] Change
test_stateful_stream_job_upgrade to use stop-with-savepoint.
Posted by kk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 2369e9c164a443a025d88162f0ac5b11702044d5
Author: Kostas Kloudas <kk...@gmail.com>
AuthorDate: Wed Jul 31 14:47:12 2019 +0200
[FLINK-13434][e2e] Change test_stateful_stream_job_upgrade to use stop-with-savepoint.
---
.../test-scripts/test_stateful_stream_job_upgrade.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh b/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh
index 0b871f5..e92b55c 100755
--- a/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh
+++ b/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh
@@ -58,10 +58,10 @@ wait_job_running ${ORIGINAL_JOB}
wait_oper_metric_num_in_records stateMap2.1 200
# take a savepoint of the state machine job
-SAVEPOINT_PATH=$(take_savepoint ${ORIGINAL_JOB} ${TEST_DATA_DIR} \
+SAVEPOINT_PATH=$(stop_with_savepoint ${ORIGINAL_JOB} ${TEST_DATA_DIR} \
| grep "Savepoint completed. Path:" | sed 's/.* //g')
-cancel_job ${ORIGINAL_JOB}
+wait_job_terminal_state "${ORIGINAL_JOB}" "FINISHED"
JOB=$(job ${NEW_DOP})
UPGRADED_JOB=$(${JOB} --test.job.variant upgraded \
@@ -71,5 +71,5 @@ wait_job_running ${UPGRADED_JOB}
wait_oper_metric_num_in_records stateMap3.2 200
-# if state is errorneous and the state machine job produces alerting state transitions,
+# if state is erroneous and the state machine job produces alerting state transitions,
# output would be non-empty and the test will not pass
[flink] 04/04: [hotfix][e2e] Harden
test_stateful_stream_job_upgrade e2e test.
Posted by kk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit a33ac4ca9f4031bbaaad8e48f7abb01e281940cc
Author: Kostas Kloudas <kk...@gmail.com>
AuthorDate: Wed Jul 31 15:35:08 2019 +0200
[hotfix][e2e] Harden test_stateful_stream_job_upgrade e2e test.
---
.../test-scripts/test_stateful_stream_job_upgrade.sh | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh b/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh
index e92b55c..1c48528 100755
--- a/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh
+++ b/flink-end-to-end-tests/test-scripts/test_stateful_stream_job_upgrade.sh
@@ -63,6 +63,17 @@ SAVEPOINT_PATH=$(stop_with_savepoint ${ORIGINAL_JOB} ${TEST_DATA_DIR} \
wait_job_terminal_state "${ORIGINAL_JOB}" "FINISHED"
+# isolate the path without the scheme ("file:") and do the necessary checks
+SAVEPOINT_DIR=${SAVEPOINT_PATH#"file:"}
+
+if [ -z "$SAVEPOINT_DIR" ]; then
+ echo "Savepoint location was empty. This may mean that the stop-with-savepoint failed."
+ exit 1
+elif [ ! -d "$SAVEPOINT_DIR" ]; then
+ echo "Savepoint $SAVEPOINT_PATH does not exist."
+ exit 1
+fi
+
JOB=$(job ${NEW_DOP})
UPGRADED_JOB=$(${JOB} --test.job.variant upgraded \
| grep "Job has been submitted with JobID" | sed 's/.* //g')
[flink] 03/04: [hotfix][e2e] Harden test_resume_savepoint e2e test.
Posted by kk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit b58007504287d8f39a7abd28550a8bda54526948
Author: Kostas Kloudas <kk...@gmail.com>
AuthorDate: Wed Jul 31 15:33:10 2019 +0200
[hotfix][e2e] Harden test_resume_savepoint e2e test.
---
flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh b/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh
index 47fdaca..4dfd92a 100755
--- a/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh
+++ b/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh
@@ -90,6 +90,17 @@ SAVEPOINT_PATH=$(stop_with_savepoint $DATASTREAM_JOB $TEST_DATA_DIR \
wait_job_terminal_state "${DATASTREAM_JOB}" "FINISHED"
+# isolate the path without the scheme ("file:") and do the necessary checks
+SAVEPOINT_DIR=${SAVEPOINT_PATH#"file:"}
+
+if [ -z "$SAVEPOINT_DIR" ]; then
+ echo "Savepoint location was empty. This may mean that the stop-with-savepoint failed."
+ exit 1
+elif [ ! -d "$SAVEPOINT_DIR" ]; then
+ echo "Savepoint $SAVEPOINT_PATH does not exist."
+ exit 1
+fi
+
# Since it is not possible to differentiate reporter output between the first and second execution,
# we remember the number of metrics sampled in the first execution so that they can be ignored in the following monitorings
OLD_NUM_METRICS=$(get_num_metric_samples)
[flink] 01/04: [FLINK-13434][e2e] Change the test_resume_savepoint
to use stop-with-savepoint.
Posted by kk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit d57ff710e361b74ba9703f670f92d039b7b9c08f
Author: Kostas Kloudas <kk...@gmail.com>
AuthorDate: Wed Jul 31 14:45:23 2019 +0200
[FLINK-13434][e2e] Change the test_resume_savepoint to use stop-with-savepoint.
---
.../src/main/java/org/apache/flink/client/cli/CliFrontend.java | 6 +++---
flink-end-to-end-tests/test-scripts/common.sh | 4 ++++
flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh | 4 ++--
3 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
index 95d52d9..6045aff 100644
--- a/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
+++ b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java
@@ -531,14 +531,14 @@ public class CliFrontend {
activeCommandLine,
commandLine,
clusterClient -> {
+ final String savepointPath;
try {
- clusterClient.stopWithSavepoint(jobId, advanceToEndOfEventTime, targetDirectory);
+ savepointPath = clusterClient.stopWithSavepoint(jobId, advanceToEndOfEventTime, targetDirectory);
} catch (Exception e) {
throw new FlinkException("Could not stop with a savepoint job \"" + jobId + "\".", e);
}
+ logAndSysout("Savepoint completed. Path: " + savepointPath);
});
-
- logAndSysout((advanceToEndOfEventTime ? "Drained job " : "Suspended job ") + "\"" + jobId + "\" with a savepoint.");
}
/**
diff --git a/flink-end-to-end-tests/test-scripts/common.sh b/flink-end-to-end-tests/test-scripts/common.sh
index 714150c..7ed17f0 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -471,6 +471,10 @@ function wait_job_terminal_state {
done
}
+function stop_with_savepoint {
+ "$FLINK_DIR"/bin/flink stop -p $2 $1
+}
+
function take_savepoint {
"$FLINK_DIR"/bin/flink savepoint $1 $2
}
diff --git a/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh b/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh
index 0a92bf5..47fdaca 100755
--- a/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh
+++ b/flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh
@@ -85,10 +85,10 @@ wait_job_running $DATASTREAM_JOB
wait_oper_metric_num_in_records SemanticsCheckMapper.0 200
# take a savepoint of the state machine job
-SAVEPOINT_PATH=$(take_savepoint $DATASTREAM_JOB $TEST_DATA_DIR \
+SAVEPOINT_PATH=$(stop_with_savepoint $DATASTREAM_JOB $TEST_DATA_DIR \
| grep "Savepoint completed. Path:" | sed 's/.* //g')
-cancel_job $DATASTREAM_JOB
+wait_job_terminal_state "${DATASTREAM_JOB}" "FINISHED"
# Since it is not possible to differentiate reporter output between the first and second execution,
# we remember the number of metrics sampled in the first execution so that they can be ignored in the following monitorings