You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by azagrebin <gi...@git.apache.org> on 2018/04/26 14:24:59 UTC
[GitHub] flink pull request #5921: [FLINK-9254] Move NotSoMiniClusterIterations to be...
GitHub user azagrebin opened a pull request:
https://github.com/apache/flink/pull/5921
[FLINK-9254] Move NotSoMiniClusterIterations to be an end-to-end test
## What is the purpose of the change
This PR moves `NotSoMiniClusterIterations` to nightly e2e tests.
## Brief change log
- submodule `flink-high-parallelism-iterations-test` added to `flink-end-to-end-tests_2.11`
- `NotSoMiniClusterIterations` moved to `HighParallelismIterationsTestProgram` in `flink-high-parallelism-iterations-test` and factored not to use MiniCluster but run as a normal flink job.
- `test_high_parallelism_iterations.sh` starts flink cluster with 100 task managers and runs `HighParallelismIterationsTestProgram`
- `test_high_parallelism_iterations.sh` is added `run-nightly-tests.sh`
## Verifying this change
- mvn clean -DskipTests package
- FLINK_DIR=build-target PARALLELISM=10 ./flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
Real test is set to run 100 task managers by default.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/azagrebin/flink FLINK-9254
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5921.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5921
----
commit 638e7233345742fb085281b202796e4e9e40e7ee
Author: Andrey Zagrebin <an...@...>
Date: 2018-04-26T13:58:09Z
[FLINK-9254] Move NotSoMiniClusterIterations to be an end-to-end test
----
---
[GitHub] flink issue #5921: [FLINK-9254] Move NotSoMiniClusterIterations to be an end...
Posted by azagrebin <gi...@git.apache.org>.
Github user azagrebin commented on the issue:
https://github.com/apache/flink/pull/5921
cc @StefanRRichter, @aljoscha
---
[GitHub] flink issue #5921: [FLINK-9254] Move NotSoMiniClusterIterations to be an end...
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/5921
In that case, LGTM 👍 Will merge this.
---
[GitHub] flink issue #5921: [FLINK-9254] Move NotSoMiniClusterIterations to be an end...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5921
should be fine, it's also only a nightly test
---
[GitHub] flink pull request #5921: [FLINK-9254] Move NotSoMiniClusterIterations to be...
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/5921#discussion_r185711958
--- Diff: flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh ---
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+source "$(dirname "$0")"/common.sh
+
+PARALLELISM="${PARALLELISM:-100}"
+
+TEST=flink-high-parallelism-iterations-test
+TEST_PROGRAM_NAME=HighParallelismIterationsTestProgram
+TEST_PROGRAM_JAR=$TEST_INFRA_DIR/../../flink-end-to-end-tests/$TEST/target/$TEST_PROGRAM_NAME.jar
+
+echo "Run Not So MiniCluster Iterations Graph Connected Components Program"
+
+cp $FLINK_DIR/conf/flink-conf.yaml $FLINK_DIR/conf/flink-conf.yaml.bak
--- End diff --
I think your job upgrade e2e test is making the backup and restore of the configuration a utility function. I suggest that we also use the function from common here once it is merged to avoid code duplication.
---
[GitHub] flink pull request #5921: [FLINK-9254] Move NotSoMiniClusterIterations to be...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5921
---