You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2015/08/21 15:59:53 UTC
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/1039
[FLINK-2386] Add new KafkaConsumer, based on Kafka's Simple Consumer API
This pull request is based on https://github.com/apache/flink/pull/1028.
I'm creating another pull request because I rebased everything to the current master.
Since there were no -1 votes #1028, I assume that this PR is close to be mergeable.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink kafka_no_copy_rebased
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1039.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1039
----
commit cbdb7d6031486b42686d0006fe1726dd58c975c3
Author: Robert Metzger <rm...@apache.org>
Date: 2015-07-20T19:39:46Z
[FLINK-2386] [kafka connector] Add new Kafka Consumer for Flink
This closes #996
commit cb0bc7c9a067881c2d0113684646f302014aca79
Author: Stephan Ewen <se...@apache.org>
Date: 2015-08-11T12:21:33Z
[streaming] Cleanup de-/serialization schema, add TypeInformationSerializationSchema prominent, add tests.
commit a62688bd7717e3d4cc79f6378209c7e4e4c83924
Author: Stephan Ewen <se...@apache.org>
Date: 2015-08-11T14:48:26Z
[FLINK-2386] [kafka connector] Add comments to all backported kafka sources and move them to 'org.apache.flink.kafka_backport'
commit 8a8003ba86f89faa15fd96fcfe6a19b54782d875
Author: Stephan Ewen <se...@apache.org>
Date: 2015-08-11T20:21:53Z
[FLINK-2386] [kafka connector] Refactor, cleanup, and fix kafka consumers
commit ac9114c76a7323c48110439e65e2f554d16c18c4
Author: Stephan Ewen <se...@apache.org>
Date: 2015-08-16T14:52:16Z
[tests] Reinforce StateCheckpoinedITCase to make sure actual checkpointing has happened before a failure.
commit 1ec61b4041dc7fb0e014a74f1cac502ae3c4e5d9
Author: Robert Metzger <rm...@apache.org>
Date: 2015-08-17T19:16:43Z
[FLINK-2386] Remove copied Kafka code again. Implemented our own topic metadata retrieval.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/1039#discussion_r37681201
--- Diff: flink-contrib/flink-storm-compatibility/flink-storm-compatibility-core/src/main/java/org/apache/flink/stormcompatibility/api/FlinkLocalCluster.java ---
@@ -41,8 +41,7 @@ public void submitTopology(final String topologyName, final Map<?, ?> conf, fina
public void submitTopologyWithOpts(final String topologyName, final Map<?, ?> conf, final FlinkTopology topology,
final SubmitOptions submitOpts) throws Exception {
- ClusterUtil
- .startOnMiniCluster(topology.getStreamGraph().getJobGraph(topologyName), topology.getNumberOfTasks());
+ ClusterUtil.startOnMiniCluster(topology.getStreamGraph().getJobGraph(topologyName), topology.getNumberOfTasks(), -1);
--- End diff --
Why is this changed? I think by default will also set it to -1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/1039#discussion_r37681294
--- Diff: flink-core/src/main/java/org/apache/flink/util/NetUtils.java ---
@@ -48,8 +48,9 @@ public static String getHostnameFromFQDN(String fqdn) {
* Works also for ipv6.
*
* See: http://stackoverflow.com/questions/2345063/java-common-way-to-validate-and-convert-hostport-to-inetsocketaddress
+ * @return URL object for accessing host and Port
*/
- public static void ensureCorrectHostnamePort(String hostPort) {
+ public static URL ensureCorrectHostnamePort(String hostPort) {
--- End diff --
Since it is now returning something, could we change the name to reflect it properly, like ``generateCorrectUrlFromHostnameAndPort``
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-134584442
I addressed @hsaputra's comments & rebased to the latest master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-135145689
Merging this...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-133478852
Damn. You are right. I'll add the full type parameters again for the 0.9.1 backport (once this one is merged).
> Should we even differentiate between them, or only port the PersistentKafkaSource to extend the FlinkKafkaConsumer
That's a good question. I though that having the two user-facing classes would allow us to "silently" change the implementation for the different Kafka versions if needed (for example when the new consumer API is out, and we know its fully backwards compatible).
But I guess this is a matter of taste. I can change the name to FlinkKafkaConsumer (without the version suffix) if you want.
Do you think having two `PersistentKafkaSource` in different packages is a good idea?
I don't know whether Sink / Source or Producer / Consumer are better pairs of names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-133483983
Concerning the `FlinkKafkaConsumer081` and `FlinkKafkaConsumer082` - lets keep them exactly to fix bugs underneath the hood.
I would actually move all this from the `flink-connector-kafka-083`to the other kafka connector project and replace the old `PersistentKafkaSource`.
In a sepaate patch, we should we add some convenience constructors, where one can directly pass the list of brokers, rather than a properties object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-133494751
Okay, will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1039
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-133470243
This needs to go into 0.9.1, so the diamond operators are going to be a problem
There is no difference, currently, between the 0.8.2 and the 0.8.1 Kafka consumers. Should we even differentiate between them, or only port the `PersistentKafkaSource` to extend the `FlinkKafkaConsumer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1039#issuecomment-133502218
Okay, I've relocated the source into the `flink-connector-kafka` module.
I added a convenience class `PersistentKafkaSource` in the exact same package as in the old module, with the same Ctor arguments. So 0.9.0 users can just set their version to 0.9.1 without touching the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---