You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tweise <gi...@git.apache.org> on 2018/02/14 00:44:36 UTC
[GitHub] flink pull request #5480: [FLINK-8648] [kinesis] Allow for customization of ...
GitHub user tweise opened a pull request:
https://github.com/apache/flink/pull/5480
[FLINK-8648] [kinesis] Allow for customization of emitRecordAndUpdateState in Kinesis connector.
## What is the purpose of the change
Allow customization of record emission in the Kinesis consumer. In this case we will use it to implement custom watermark logic but it could also be used for any other logic that inspects the record with contextual information about the shard.
## Brief change log
- Trivial change to remove final from the declaration to enable override.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
I didn't add a test, but perhaps we can add a NOOP example with override in the test package just to have the compile path covered?
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / *no*)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / *no*)
- The serializers: (yes / *no* / don't know)
- The runtime per-record code paths (performance sensitive): (yes / *no* / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / *no* / don't know)
- The S3 file system connector: (yes / *no* / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / *no*)
- If yes, how is the feature documented? (*not applicable* / docs / JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tweise/flink FLINK-8648.emitRecordAndUpdateState
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5480.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5480
----
commit c9b6369796cfa03448a1da32313981c581304a4b
Author: Thomas Weise <th...@...>
Date: 2018-02-14T00:33:59Z
[FLINK-8648] [kinesis] Allow for customization of emitRecordAndUpdateState in Kinesis connector.
----
---
[GitHub] flink issue #5480: [FLINK-8648] [kinesis] Allow for customization of emitRec...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/5480
I see, so it is basically a quicker workaround for per-shard watermarks for the Kinesis Consumer.
Since we discussed that we'll likely have a reworked consumer in 1.6 that hopefully includes per-shard watermarks, I'll agree to merge this temporary workaround change now for 1.5.
---
[GitHub] flink issue #5480: [FLINK-8648] [kinesis] Allow for customization of emitRec...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the issue:
https://github.com/apache/flink/pull/5480
@tzulitai any update?
---
[GitHub] flink issue #5480: [FLINK-8648] [kinesis] Allow for customization of emitRec...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the issue:
https://github.com/apache/flink/pull/5480
@tzulitai see https://gist.github.com/tweise/7fad5d5df0abf54670a52d0d02d61179 for details.
As indicated in the email thread, emit override will track watermark state and delegate the record to the base implementation.
---
[GitHub] flink pull request #5480: [FLINK-8648] [kinesis] Allow for customization of ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5480
---
[GitHub] flink issue #5480: [FLINK-8648] [kinesis] Allow for customization of emitRec...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the issue:
https://github.com/apache/flink/pull/5480
R: @tzulitai
---
[GitHub] flink issue #5480: [FLINK-8648] [kinesis] Allow for customization of emitRec...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/5480
Just curious, before I proceed to merge this:
could you briefly describe what your custom override would consist of?
I'm asking because we may be able to find a cleaner solution (or not 😅). The method was made final, because that part is critical for exactly-once guarantees.
It would also be interesting to get to know what you are trying to achieve here.
---