You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by florianschmidt1994 <gi...@git.apache.org> on 2018/06/15 09:21:32 UTC
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
GitHub user florianschmidt1994 opened a pull request:
https://github.com/apache/flink/pull/6172
[FLINK-9594][E2E tests] Improve docs for new test runner]
## What is the purpose of the change
Improve documentation for end-to-end tests with the new test runner pattern introduced in FLINK-9257
## Brief change log
- Add more docs
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/florianschmidt1994/flink FLINK-9594-add-docs-for-e2e-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6172.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6172
----
commit 7a0761ac27d5b876425616cc9433daa0be949cc9
Author: Florian Schmidt <fl...@...>
Date: 2018-06-15T09:13:39Z
[FLINK-9594][E2E tests] Improve docs for new test runner]
----
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195992732
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible, try
-to put new functionality in common.sh so that it can be reused by other tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
+to put new functionality in `common.sh` so that it can be reused by other tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
+A test is considered to have failed if it:
+- exited with non-zero exit code
+- has non-empty *.out files (something was written to stdout / stderr by your flink program)
+- there are errors in the log files
+- there are exceptions in the log files
+
+_There is a whitelist for exceptions and errors that do not lead to failure, they can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_
+
+Please note that a previously supported pattern where you could assign a value the global variable `PASS` to have your tests fail **is not supported anymore**
--- End diff --
Missing period at the end.
---
[GitHub] flink issue #6172: [FLINK-9594][E2E tests] Improve docs for new test runner]
Posted by florianschmidt1994 <gi...@git.apache.org>.
Github user florianschmidt1994 commented on the issue:
https://github.com/apache/flink/pull/6172
@tzulitai or @zentol Can you skim over this if you have some time?
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195992679
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible, try
-to put new functionality in common.sh so that it can be reused by other tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
+to put new functionality in `common.sh` so that it can be reused by other tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
+A test is considered to have failed if it:
+- exited with non-zero exit code
+- has non-empty *.out files (something was written to stdout / stderr by your flink program)
+- there are errors in the log files
+- there are exceptions in the log files
+
+_There is a whitelist for exceptions and errors that do not lead to failure, they can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_
--- End diff --
`... not lead to failure, they can be found ...` --> `.. not lead to failure, "which" can be found`
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195823151
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible, try
-to put new functionality in common.sh so that it can be reused by other tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
+to put new functionality in `common.sh` so that it can be reused by other tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
--- End diff --
refer to the templates in the respective files instead
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195992920
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible, try
-to put new functionality in common.sh so that it can be reused by other tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
+to put new functionality in `common.sh` so that it can be reused by other tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
+A test is considered to have failed if it:
+- exited with non-zero exit code
+- has non-empty *.out files (something was written to stdout / stderr by your flink program)
+- there are errors in the log files
+- there are exceptions in the log files
+
+_There is a whitelist for exceptions and errors that do not lead to failure, they can be found in the `check_logs_for_errors` and `check_logs_for_exceptions` in `test-scripts/common.sh`_
+
+Please note that a previously supported pattern where you could assign a value the global variable `PASS` to have your tests fail **is not supported anymore**
+
+### Cleanup
+The test runner performs a cleanup after each test case, which includes:
+- Stopping the cluster
+- Killing all task and job managers
+- Reverting config to default (if changed before)
+- Cleaning up log and temp directories
+
+If your test case should do some *additional* cleanup, a common pattern is to trap a `test_cleanup` function to `EXIT` in your test case like so:
--- End diff --
I would explicitly state that this typically consists of shutting down external systems used by the e2e test, e.g. Kafka / Elasticsearch.
---
[GitHub] flink issue #6172: [FLINK-9594][E2E tests] Improve docs for new test runner
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/6172
Thanks @florianschmidt1994, merging this ..
---
[GitHub] flink issue #6172: [FLINK-9594][E2E tests] Improve docs for new test runner
Posted by florianschmidt1994 <gi...@git.apache.org>.
Github user florianschmidt1994 commented on the issue:
https://github.com/apache/flink/pull/6172
Thanks for proof-reading! I addressed your comments and squashed the PR
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195991753
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible, try
-to put new functionality in common.sh so that it can be reused by other tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
+to put new functionality in `common.sh` so that it can be reused by other tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program)
--- End diff --
Capital `f` for `Flink`
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/6172
---
[GitHub] flink pull request #6172: [FLINK-9594][E2E tests] Improve docs for new test ...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6172#discussion_r195823386
--- Diff: flink-end-to-end-tests/README.md ---
@@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> flink-end-to-end-tests/test-scripts/test_batch_wordcount
## Writing Tests
-Have a look at test_batch_wordcount.sh for a very basic test and
-test_streaming_kafka010.sh for a more involved example. Whenever possible, try
-to put new functionality in common.sh so that it can be reused by other tests.
+### Examples
+Have a look at `test_batch_wordcount.sh` for a very basic test and
+`test_streaming_kafka010.sh` for a more involved example. Whenever possible, try
+to put new functionality in `common.sh` so that it can be reused by other tests.
+
+### Adding a test case
+In order to add a new test case you need a new line to either `test-scripts/run-nightly-tests.sh` and / or `test-scripts/run-pre-commit-tests.sh`
+
+Adding a new test case generally follows the following pattern
+
+```sh
+run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh arg1 arg2"
+```
+
+_Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually._
+
+### Passing your test
+A test is considered to have passed if it:
+- has exit code 0
+- there are no non-empty .out files (nothing was written to stdout / stderr by your flink program)
+- there are no exceptions in the log files
+- there are no errors in the log files
+
+### Failing your test
--- End diff --
this section is just the inverse of `Passing your test`, we can omit it.
---