You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tzulitai <gi...@git.apache.org> on 2018/03/22 09:15:00 UTC
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/5745
[FLINK-8976] [test] Add end-to-end test for resuming savepoints with different parallelism
## What is the purpose of the change
This PR adds end-to-end tests for resuming a savepoint with different parallelisms.
## Brief change log
The changes are based on the new `test_resume_savepoint.sh` test script in #5733, so only the last commit is relevant.
That script has been modified to be able to specify parallelism of the state machine job, before and after the savepoint restore.
- Adapt `test_resume_savepoint.sh` to be able to use different parallelism after the savepoint restore.
- Add scale up / scale down tests to be executed by Travis.
## Verifying this change
This PR adds new tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-8976
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5745.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5745
----
commit 529e060cb05fd723b8656dcc9ef48f8011282dd8
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-03-21T08:25:37Z
[FLINK-8975] [test] Add Kafka events generator job for StateMachineExample
commit 6b5126c006c752c6c1bee2699d429927e74587c9
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-03-21T08:32:51Z
[FLINK-8975] [test] Add resume from savepoint end-to-end test
commit 475ef4de0f8abbd3bb485e0e6fefcce2612074d3
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-03-22T07:34:49Z
fixup! Use SLF4J reporter to monitor state machine progress
commit 55d69567bc4cb61cd08ad852126c9c2c39b8feb4
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2018-03-22T09:09:53Z
[FLINK-8976] [test] Add end-to-end tests for resuming savepoint with differrent parallelism
----
---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5745
---
[GitHub] flink issue #5745: [FLINK-8976] [test] Add end-to-end test for resuming save...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/5745
Thanks @zentol.
I'll merge this (as well as #5733) with your comments addressed, as soon as Travis shows green.
---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/5745#discussion_r176678540
--- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh ---
@@ -17,11 +17,25 @@
# limitations under the License.
################################################################################
+if [ -z $1 ] || [ -z $2 ]; then
+ echo "Usage: ./test_resume_savepoint.sh <original_dop> <new_dop>"
+ exit 1
+fi
+
source "$(dirname "$0")"/common.sh
-# modify configuration to have 2 slots
+ORIGINAL_DOP=$1
+NEW_DOP=$2
+
+if (( $ORIGINAL_DOP >= $NEW_DOP )); then
+ NUM_SLOTS=$(( $ORIGINAL_DOP + 1 ))
+else
+ NUM_SLOTS=$(( $NEW_DOP + 1 ))
+fi
+
+# modify configuration to have enough slots
cp $FLINK_DIR/conf/flink-conf.yaml $FLINK_DIR/conf/flink-conf.yaml.bak
-sed -i -e 's/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: 2/' $FLINK_DIR/conf/flink-conf.yaml
+sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $NUM_SLOTS/" $FLINK_DIR/conf/flink-conf.yaml
# modify configuration to use SLF4J reporter; we will be using this to monitor the state machine progress
cp $FLINK_DIR/opt/flink-metrics-slf4j-1.6-SNAPSHOT.jar $FLINK_DIR/lib/
--- End diff --
That makes sense, will fix this.
---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5745#discussion_r176677259
--- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh ---
@@ -17,11 +17,25 @@
# limitations under the License.
################################################################################
+if [ -z $1 ] || [ -z $2 ]; then
+ echo "Usage: ./test_resume_savepoint.sh <original_dop> <new_dop>"
+ exit 1
+fi
+
source "$(dirname "$0")"/common.sh
-# modify configuration to have 2 slots
+ORIGINAL_DOP=$1
+NEW_DOP=$2
+
+if (( $ORIGINAL_DOP >= $NEW_DOP )); then
+ NUM_SLOTS=$(( $ORIGINAL_DOP + 1 ))
+else
+ NUM_SLOTS=$(( $NEW_DOP + 1 ))
+fi
+
+# modify configuration to have enough slots
cp $FLINK_DIR/conf/flink-conf.yaml $FLINK_DIR/conf/flink-conf.yaml.bak
-sed -i -e 's/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: 2/' $FLINK_DIR/conf/flink-conf.yaml
+sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $NUM_SLOTS/" $FLINK_DIR/conf/flink-conf.yaml
# modify configuration to use SLF4J reporter; we will be using this to monitor the state machine progress
cp $FLINK_DIR/opt/flink-metrics-slf4j-1.6-SNAPSHOT.jar $FLINK_DIR/lib/
--- End diff --
can we define the version as an environment variable in the common.sh?
---
[GitHub] flink issue #5745: [FLINK-8976] [test] Add end-to-end test for resuming save...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5745
we may improve the test times by writing a custom reporter specifically for this test that writes to the log if the condition is met.
---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5745#discussion_r176677065
--- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh ---
@@ -17,11 +17,25 @@
# limitations under the License.
################################################################################
+if [ -z $1 ] || [ -z $2 ]; then
+ echo "Usage: ./test_resume_savepoint.sh <original_dop> <new_dop>"
+ exit 1
+fi
+
source "$(dirname "$0")"/common.sh
-# modify configuration to have 2 slots
+ORIGINAL_DOP=$1
+NEW_DOP=$2
+
+if (( $ORIGINAL_DOP >= $NEW_DOP )); then
+ NUM_SLOTS=$(( $ORIGINAL_DOP + 1 ))
--- End diff --
would be good to explain where the +1 comes from. (i guess it is for the kafka job)
---
[GitHub] flink pull request #5745: [FLINK-8976] [test] Add end-to-end test for resumi...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/5745#discussion_r176678588
--- Diff: flink-end-to-end-tests/test-scripts/test_resume_savepoint.sh ---
@@ -17,11 +17,25 @@
# limitations under the License.
################################################################################
+if [ -z $1 ] || [ -z $2 ]; then
+ echo "Usage: ./test_resume_savepoint.sh <original_dop> <new_dop>"
+ exit 1
+fi
+
source "$(dirname "$0")"/common.sh
-# modify configuration to have 2 slots
+ORIGINAL_DOP=$1
+NEW_DOP=$2
+
+if (( $ORIGINAL_DOP >= $NEW_DOP )); then
+ NUM_SLOTS=$(( $ORIGINAL_DOP + 1 ))
--- End diff --
👌 Yes, it is for the Kafka event generator job.
---