You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by florianschmidt1994 <gi...@git.apache.org> on 2018/03/26 10:29:43 UTC
[GitHub] flink pull request #5762: [FLINK-9078][E2ETests] Add test that verifies corr...
GitHub user florianschmidt1994 opened a pull request:
https://github.com/apache/flink/pull/5762
[FLINK-9078][E2ETests] Add test that verifies correct avro behaviour
## What is the purpose of the change
There was a specific kind of classloading issue where a flink job would fail with a ClassCastException when restarting after failure, which has now been fixed.
This end2end test adds automated testing to verify that this error does not occur anymore.
## Brief change log
- Add example module with avro based on quickstart example
- Add test script
- Add utility functions and improve output in `common.sh`
## Verifying this change
This change added tests and can be verified as follows:
Run `./run-pre-commit-tests.sh`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/florianschmidt1994/flink avro-end-to-end-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5762.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5762
----
commit 27235527a6cdc9d3ea148f639dbf98e4af2c3f8b
Author: Florian Schmidt <fl...@...>
Date: 2018-03-21T13:50:21Z
Add avro e2e test
commit 69d7a38b6744ffda86cedb274c19cdcda96e9f17
Author: Florian Schmidt <fl...@...>
Date: 2018-03-26T10:29:00Z
Add license header
----
---
[GitHub] flink pull request #5762: [FLINK-9078][E2ETests] Add test that verifies corr...
Posted by florianschmidt1994 <gi...@git.apache.org>.
Github user florianschmidt1994 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5762#discussion_r178828282
--- Diff: flink-end-to-end-tests/flink-avro-classloading-test/src/main/avro/user.avsc ---
@@ -0,0 +1,9 @@
+{
+ "type": "record",
+ "namespace": "org.apache.flink.tests.streaming",
+ "name": "User",
+ "fields": [
+ { "name": "first", "type": "string" },
+ { "name": "last", "type": "string" }
--- End diff --
I added some more complex examples and verified that the initial test case still fails on 1.3.X and works on 1.4.X as expected
---
[GitHub] flink issue #5762: [FLINK-9078][E2ETests] Add test that verifies correct avr...
Posted by florianschmidt1994 <gi...@git.apache.org>.
Github user florianschmidt1994 commented on the issue:
https://github.com/apache/flink/pull/5762
@aljoscha @patricklucas Could you maybe have a look at this?
---
[GitHub] flink pull request #5762: [FLINK-9078][E2ETests] Add test that verifies corr...
Posted by florianschmidt1994 <gi...@git.apache.org>.
Github user florianschmidt1994 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5762#discussion_r178758895
--- Diff: flink-end-to-end-tests/flink-avro-classloading-test/src/main/avro/user.avsc ---
@@ -0,0 +1,9 @@
+{
+ "type": "record",
+ "namespace": "org.apache.flink.tests.streaming",
+ "name": "User",
+ "fields": [
+ { "name": "first", "type": "string" },
+ { "name": "last", "type": "string" }
--- End diff --
I can do that. The initial idea was to add this e2e test for a specific failure case that got fixed in 1.4 something, but there shouldn't be a downside to extending it with more complex fields, should it?
---
[GitHub] flink pull request #5762: [FLINK-9078][E2ETests] Add test that verifies corr...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5762#discussion_r177059068
--- Diff: flink-end-to-end-tests/flink-avro-classloading-test/src/main/avro/user.avsc ---
@@ -0,0 +1,9 @@
+{
+ "type": "record",
+ "namespace": "org.apache.flink.tests.streaming",
+ "name": "User",
+ "fields": [
+ { "name": "first", "type": "string" },
+ { "name": "last", "type": "string" }
--- End diff --
Could you add a more complex Avro type? I think we should test all available Avro types. Esp. maps and arrays and maybe different `SpecificRecords` and `GenericRecords`.
---
[GitHub] flink issue #5762: [FLINK-9078][E2ETests] Add test that verifies correct avr...
Posted by florianschmidt1994 <gi...@git.apache.org>.
Github user florianschmidt1994 commented on the issue:
https://github.com/apache/flink/pull/5762
@twalthr How are we planning to proceed on this PR?
---