You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/05/22 12:29:00 UTC
[jira] [Commented] (FLINK-9257) End-to-end tests prints "All tests
PASS" even if individual test-script returns non-zero exit code
[ https://issues.apache.org/jira/browse/FLINK-9257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16483867#comment-16483867 ]
ASF GitHub Bot commented on FLINK-9257:
---------------------------------------
GitHub user florianschmidt1994 opened a pull request:
https://github.com/apache/flink/pull/6053
[FLINK-9257][E2E Tests] Fix wrong "All tests pass" message
## What is the purpose of the change
Fix a wrongly printed "All tests PASS" message when they actually don't"
Previously the behaviour was like this:
>During the cleanup hook (trap cleanup EXIT in common.sh) it will be checked whether there are non-empty out files or log files with certain exceptions. If a tests fails with non-zero exit code, but does not have any exceptions or .out files, this will still print "All tests PASS" to stdout, even though they don't
With this PR the whole test-runner is restructured so that
1. The check for non-empty .out files, errors and exceptions in logs is triggered from the `run_test` method
2. The error message after each test is dependant on both the exit code of the test script as well as the result from checking the log files
3. cleanup is now triggered by the test runner, not by the individual tests anymore
4. tests that signaled their failure by modifying `PASS` now do so by exiting with non-zero exit code
5. `check_result_hash` exits with 1 instead of modifying `PASS`
Additionally this PR
1. Reformats the output a little compared to previous tests
```
Flink dist directory: /Users/florianschmidt/dev/flink/flink-dist/target/flink-1.6-SNAPSHOT-bin/flink-1.6-SNAPSHOT
TEST_DATA_DIR: /Users/florianschmidt/dev/flink/flink-end-to-end-tests/temp-test-directory-00N
Flink dist directory: /Users/florianschmidt/dev/flink/flink-dist/target/flink-1.6-SNAPSHOT-bin/flink-1.6-SNAPSHOT
TEST_DATA_DIR: /Users/florianschmidt/dev/flink/flink-end-to-end-tests/temp-test-directory-00N
flink-end-to-end-test directory: /Users/florianschmidt/dev/flink/flink-end-to-end-tests
Flink distribution directory: /Users/florianschmidt/dev/flink/flink-dist/target/flink-1.6-SNAPSHOT-bin/flink-1.6-SNAPSHOT
==============================================================================
Running 'Streaming Python Wordcount end-to-end test'
==============================================================================
Flink dist directory: /Users/florianschmidt/dev/flink/flink-dist/target/flink-1.6-SNAPSHOT-bin/flink-1.6-SNAPSHOT
TEST_DATA_DIR: /Users/florianschmidt/dev/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-00N
Starting cluster.
Starting standalonesession daemon on host Florians-MBP.fritz.box.
Starting taskexecutor daemon on host Florians-MBP.fritz.box.
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
Waiting for dispatcher REST endpoint to come up...
Dispatcher REST endpoint is up.
Starting execution of program
Program execution finished
Job with JobID 436dfd1f2a81ab4f818fc7fb9c395f0c has finished.
Job Runtime: 7512 ms
pass StreamingPythonWordCount
Stopping taskexecutor daemon (pid: 9877) on host Florians-MBP.fritz.box.
Stopping standalonesession daemon (pid: 9585) on host Florians-MBP.fritz.box.
No zookeeper daemon to stop on host Florians-MBP.fritz.box.
[PASS] 'Streaming Python Wordcount end-to-end test' passed after 0 minutes and 22 seconds! Test exited with exit code 0.
==============================================================================
Running 'Wordcount end-to-end test'
==============================================================================
Flink dist directory: /Users/florianschmidt/dev/flink/flink-dist/target/flink-1.6-SNAPSHOT-bin/flink-1.6-SNAPSHOT
TEST_DATA_DIR: /Users/florianschmidt/dev/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-24N
```
## Verifying this change
* I ran the test scripts manually and checked that they still behave as expected
* I used the following script as a sample e2e-test to trigger different failure / success behaviours
```
#!/usr/bin/env bash
source "$(dirname "$0")"/common.sh
# each of those can be used to cause a test to fail
# echo "This should cause the test to fail" > $FLINK_DIR/log/test.out
# check_result_hash "asf" "$FLINK_DIR/log/"
# exit 1
function test_cleanup {
echo "Something"
# Uncomment to see test fail in cleanup
# exit 2
}
trap test_cleanup EXIT
```
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/florianschmidt1994/flink flink-9257-fix-all-tests-pass-message
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6053.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6053
----
commit 2ca08c6ed0a4faea7e4094004d8d73aaa9140ac3
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T09:36:55Z
Move error/exception/out checking to functions
commit 54c23a7858efad8d2b7dfda932ad5f50080bf95a
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T11:19:23Z
Use changed test runner for nightly tests
commit 9cac304e4852b1c89ccf750446bda667768be2bb
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T11:22:03Z
Use new test runner for pre-commit tests
commit c5108fee328e4b33558c46a6f52e03a2aa7a6dbc
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T11:35:17Z
Use script exit codes where PASS="" has been used
commit ee2cfc489209baa8963adbdc05c55bf6b8f41aae
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T11:44:29Z
Exit immediately on check_result_hash failure
commit acbbae63e11fceac74e1e691e967731235268a8c
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T11:45:02Z
Add missing initialization for test_has_errors
commit 43a6244e34dcbdfa9727b49b0946b0f055510cf9
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T11:59:55Z
Restructure test runner for nicer output
commit d4785eb23bf772cded6859f0545812a1be1e0991
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T12:03:38Z
Remove sample test and unnecessary EXIT_CODE
commit 8f404f9f6dedbf58ab96fa515569ea50b83d79f7
Author: Florian Schmidt <fl...@...>
Date: 2018-05-22T12:11:41Z
Add all tests passed message
----
> End-to-end tests prints "All tests PASS" even if individual test-script returns non-zero exit code
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-9257
> URL: https://issues.apache.org/jira/browse/FLINK-9257
> Project: Flink
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.5.0
> Reporter: Florian Schmidt
> Assignee: Florian Schmidt
> Priority: Critical
> Fix For: 1.5.0
>
>
> In some cases the test-suite exits with non-zero exit code but still prints "All tests PASS" to stdout. This happens because how the test runner works, which is roughly as follows
> # Either run-nightly-tests.sh or run-precommit-tests.sh executes a suite of tests consisting of one multiple bash scripts.
> # As soon as one of those bash scripts exists with non-zero exit code, the tests won't continue to run and the test-suite will also exit with non-zero exit code.
> # *During the cleanup hook (trap cleanup EXIT in common.sh) it will be checked whether there are non-empty out files or log files with certain exceptions. If a tests fails with non-zero exit code, but does not have any exceptions or .out files, this will still print "All tests PASS" to stdout, even though they don't*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)