You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by mylog00 <gi...@git.apache.org> on 2017/02/06 13:13:00 UTC
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
GitHub user mylog00 opened a pull request:
https://github.com/apache/flink/pull/3270
[FLINK-4286] Have Kafka examples that use the Kafka 0.9 connector
There were added examples of using connector for Kafka v0.9. Examples using 0.8 connector renamed to "ReadFromKafka08" and "WriteIntoKafka08" respectively. The pom.xml was modified: added config for separated packages Kafka08.jar and Kafka09.jar using "maven-jar-plugin" instead "maven-shade-plugin".
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mylog00/flink FLINK-4286
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3270.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3270
----
commit fc3498f5db15c4a5685af8dc96728de07f852287
Author: Dmitrii Kniazev <dm...@epam.com>
Date: 2017-02-06T08:58:52Z
[FLINK-4286] Have Kafka examples that use the Kafka 0.9 connector
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 commented on the issue:
https://github.com/apache/flink/pull/3270
Hi @rmetzger, i delete example for Kafka v0.8 and left only for Kafka v0.9.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3270#discussion_r124046763
--- Diff: flink-examples/flink-examples-streaming/pom.xml ---
@@ -482,54 +502,19 @@ under the License.
<copy file="${project.basedir}/target/flink-examples-streaming_2.10-${project.version}-WindowJoin.jar" tofile="${project.basedir}/target/WindowJoin.jar" />
<copy file="${project.basedir}/target/flink-examples-streaming_2.10-${project.version}-WordCount.jar" tofile="${project.basedir}/target/WordCount.jar" />
<copy file="${project.basedir}/target/flink-examples-streaming_2.10-${project.version}-SocketWindowWordCount.jar" tofile="${project.basedir}/target/SocketWindowWordCount.jar" />
+ <copy file="${project.basedir}/target/flink-examples-streaming_2.10-${project.version}-Kafka09.jar" tofile="${project.basedir}/target/Kafka09.jar" />
</target>
</configuration>
</execution>
</executions>
</plugin>
- <!-- Use the shade plugin to build a fat jar for the kafka example -->
+ <!-- Use the shade plugin to build a fat jar for the twitter example -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
- <id>fat-jar-kafka-example</id>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <shadeTestJar>false</shadeTestJar>
- <shadedArtifactAttached>false</shadedArtifactAttached>
- <createDependencyReducedPom>false</createDependencyReducedPom>
- <transformers>
- <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <mainClass>org.apache.flink.streaming.examples.kafka.ReadFromKafka</mainClass>
- </transformer>
- </transformers>
- <finalName>Kafka</finalName>
- <!-- <outputFile>Kafka.jar</outputFile> -->
- <filters>
- <filter>
- <artifact>*</artifact>
- <includes>
- <include>org/apache/flink/streaming/examples/kafka/**</include>
- <include>org/apache/flink/streaming/**</include>
- <include>org/apache/kafka/**</include>
- <include>org/apache/curator/**</include>
- <include>org/apache/zookeeper/**</include>
- <include>org/apache/jute/**</include>
- <include>org/I0Itec/**</include>
- <include>jline/**</include>
- <include>com/yammer/**</include>
- <include>kafka/**</include>
- </includes>
- </filter>
- </filters>
- </configuration>
- </execution>
- <execution>
--- End diff --
What's the reasoning for deleting this section?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 commented on the issue:
https://github.com/apache/flink/pull/3270
Many merge conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3270
Hi @mylog00, thanks a lot for your contribution.
I wonder if it makes more sense to just upgrade the existing Kafka examples to use Kafka 0.9 instead of having two examples that are almost identical?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 commented on the issue:
https://github.com/apache/flink/pull/3270
I did examples more similar. Now difference only Kafka connector version and number of properties to configure consumer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 commented on the issue:
https://github.com/apache/flink/pull/3270
https://travis-ci.org/mylog00/flink/builds/198855164
Link to my TravisCI build of branch [FLINK-4286]. With all passed tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3270
Thank you for the contribution @mylog00!
It isn't necessary, but I agree with @zentol that it would be nice to have them consistent with each other :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3270#discussion_r124046687
--- Diff: flink-examples/flink-examples-streaming/pom.xml ---
@@ -360,6 +360,26 @@ under the License.
</configuration>
</execution>
+ <!--Kafka09-->
+ <execution>
+ <id>Kafka09</id>
+ <phase>package</phase>
+ <goals>
+ <goal>jar</goal>
+ </goals>
+ <configuration>
+ <classifier>Kafka09</classifier>
+ <archive>
+ <manifestEntries>
+ <program-class>org.apache.flink.streaming.examples.kafka.ReadFromKafka09</program-class>
+ </manifestEntries>
+ </archive>
+ <includes>
+ <include>org/apache/flink/streaming/examples/kafka/ReadFromKafka09.class</include>
+ <include>org/apache/flink/streaming/examples/kafka/WriteToKafka09.class</include>
+ </includes>
+ </configuration>
+ </execution>
--- End diff --
I don't think you can use the maven jar plugin for creating the kafka09 example. This won't include the needed dependencies into the jar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 closed the pull request at:
https://github.com/apache/flink/pull/3270
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3270
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by DmytroShkvyra <gi...@git.apache.org>.
Github user DmytroShkvyra commented on the issue:
https://github.com/apache/flink/pull/3270
Hi @rmetzger could you review this again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3270
This pull request actually updates the Kafka connector examples, so I think there's no need to do anything here: https://github.com/apache/flink/pull/3911/files
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 commented on the issue:
https://github.com/apache/flink/pull/3270
I think it is independent examples and make it so similar not necessary. But if it important I can fix it and decrease difference. Thank you for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3270: [FLINK-4286] Have Kafka examples that use the Kafk...
Posted by mylog00 <gi...@git.apache.org>.
Github user mylog00 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3270#discussion_r125331370
--- Diff: flink-examples/flink-examples-streaming/pom.xml ---
@@ -360,6 +360,26 @@ under the License.
</configuration>
</execution>
+ <!--Kafka09-->
+ <execution>
+ <id>Kafka09</id>
+ <phase>package</phase>
+ <goals>
+ <goal>jar</goal>
+ </goals>
+ <configuration>
+ <classifier>Kafka09</classifier>
+ <archive>
+ <manifestEntries>
+ <program-class>org.apache.flink.streaming.examples.kafka.ReadFromKafka09</program-class>
+ </manifestEntries>
+ </archive>
+ <includes>
+ <include>org/apache/flink/streaming/examples/kafka/ReadFromKafka09.class</include>
+ <include>org/apache/flink/streaming/examples/kafka/WriteToKafka09.class</include>
+ </includes>
+ </configuration>
+ </execution>
--- End diff --
Yes. My bad. I did not check what examples can be executed via command line. And I shouldn't have had to remove fat-jar packager from pom.xml file (see next comment). But this PR has many conflicts, so I close It and will create new one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3270: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3270
It would be neat if the example were more similar. Looking at the diff between the 0.8 and 0.9 examples there should only be 4-5 different lines (basically replace 08 by 09), but there are quite a few more due to formatting and such.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---