You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by kailashhd <gi...@git.apache.org> on 2018/03/14 16:26:33 UTC
[GitHub] flink pull request #5698: [FLINK-8945] [kinesis] Allow customization of Kine...
GitHub user kailashhd opened a pull request:
https://github.com/apache/flink/pull/5698
[FLINK-8945] [kinesis] Allow customization of KinesisProxy
## What is the purpose of the change
Allow customization of KinesisProxy. In this case we want to override just the getShardList implementation to use ListShards instead of DescribeStreams.
## Brief change log
- Changed the constructor from private to protected for KinesisProxy
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Also verified with mvn clean verify.
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (100MB)*
- *Extended integration test for recovery after master (JobManager) failure*
- *Added test that validates that TaskInfo is transferred only once across recoveries*
- *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kailashhd/flink KinesisProxy
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5698.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5698
----
commit 3645cdbad2b7f86eded7f0480606fd9f9fe03ec7
Author: Kailash HD <kd...@...>
Date: 2018-03-14T16:20:12Z
[FLINK-8945] [kinesis] Allow customization of KinesisProxy
----
---
[GitHub] flink issue #5698: [FLINK-8945] [kinesis] Allow customization of KinesisProx...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the issue:
https://github.com/apache/flink/pull/5698
+1
@StephanEwen can we get this into 1.5.0 ?
---
[GitHub] flink pull request #5698: [FLINK-8945] [kinesis] Allow customization of Kine...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5698
---
[GitHub] flink issue #5698: [FLINK-8945] [kinesis] Allow customization of KinesisProx...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/5698
I'm curious why we don't go directly for replacing `describeStreams` with `listShards`. Are there any other concerns for this?
---
[GitHub] flink issue #5698: [FLINK-8945] [kinesis] Allow customization of KinesisProx...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the issue:
https://github.com/apache/flink/pull/5698
@StephanEwen thanks for unblocking our work!
---
[GitHub] flink issue #5698: [FLINK-8945] [kinesis] Allow customization of KinesisProx...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5698
I think this trivial change can be safely merged into 1.5
---
[GitHub] flink issue #5698: [FLINK-8945] [kinesis] Allow customization of KinesisProx...
Posted by kailashhd <gi...@git.apache.org>.
Github user kailashhd commented on the issue:
https://github.com/apache/flink/pull/5698
cc @tweise
---