You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by mjsax <gi...@git.apache.org> on 2015/08/20 17:46:03 UTC

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

GitHub user mjsax opened a pull request:

    https://github.com/apache/flink/pull/1037

    [FLINK-2553] Example Jars not build correctly

      - reworked package structure of utils and wordcount.stormoperators package
      - reword class hierarchy of *FileSpout and *InMemorySpout
      - fixed pom.xml to assembly jars correctly
      - extended README and documenation to building and using correct jars
    additional minor changes:
     - comment typo in FlinkSubmitter
     - fixed wrong version number in hardcoded jar file name

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mjsax/flink flink-2553-examplesCleanup

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1037.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1037
    
----
commit 2b2336e02a0370e2acda9219f97eec1999dfdc46
Author: mjsax <mj...@informatik.hu-berlin.de>
Date:   2015-08-20T14:07:09Z

    [FLINK-2553] Example Jars not build correctly
      - reworked package structure of utils and wordcount.stormoperators package
      - reword class hierarchy of *FileSpout and *InMemorySpout
      - fixed pom.xml to assembly jars correctly
      - extended README and documenation to building and using correct jars
    additional minor changes:
     - comment typo in FlinkSubmitter
     - fixed wrong version number in hardcoded jar file name

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-133477238
  
    I got it now. Can be merged from my point of view.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135370943
  
    Okay, this seems related to the Kafka fix. Looking into it...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135035210
  
    Just rebased. Please merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135378423
  
    Thx. I tried to reproduce the error but without success. Also rebased to latest master and run `mvn verify` in `flink-storm-compatibility-examples` module multiple times. No error occurred.
    
    The changes in this PR aren't even related to the failing tests you observed. How do you know that it's related to the Kafka fix? Kafka is not used in the failing test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1037#discussion_r37687901
  
    --- Diff: flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/wordcount/StormWordCountRemoteBySubmitter.java ---
    @@ -71,7 +71,7 @@ public static void main(final String[] args) throws Exception {
     		// conf.put(Config.NIMBUS_THRIFT_PORT, new Integer(6123));
     
     		// The user jar file must be specified via JVM argument if executed via Java.
    -		// => -Dstorm.jar=target/flink-storm-examples-0.9-SNAPSHOT-WordCountStorm.jar
    +		// => -Dstorm.jar=target/flink-storm-examples-0.10-SNAPSHOT-WordCountStorm.jar
     		// If bin/flink is used, the jar file is detected automatically.
    --- End diff --
    
    This is a bug. I just fixed it. The jar file was renamed to "WordCount-StormTopolgy.jar". There is no version number any more. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135121906
  
    Looks fine, will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on a diff in the pull request:

https://github.com/apache/flink/pull/1037#discussion_r37573869

--- Diff: docs/apis/storm_compatibility.md ---
@@ -224,4 +224,12 @@ DataStream<String> rawInput = env.addSource(

You can find more examples in Maven module `flink-storm-compatibilty-examples`.
For the different versions of WordCount, see [README.md](https://github.com/apache/flink/tree/master/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/README.md).
+To run the examples, you need to assemble a correct jar file.
+`flink-storm-compatibility-examples-0.10-SNAPSHOT.jar` is **no** valid jar file for job execution (it is only a standard maven artifact).

+There are example jars for embedded Spout and Bolt, namely `flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountSpoutSource.jar` and `flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountBoltTokenizer.jar`, respectively.
+Compare `pom.xml` to see how both jars are built.
+Furthermore, there is one example for whole Storm topologies (`flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar`).
+This jar is build using `maven-assembly-plugin` because it must contain `defaults.yaml` which cannot be included using `maven-jar-plugin`.
--- End diff --

Can you explain or give a pointer? I was not able to include defaults.yaml without assembly-plugin?

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135342844
  
    I just saw that the header in `FiniteStormFileSpout.java` got reformatted what should not be. Can you fix this when merging?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135360613
  
    Hmmm... I will have a look. Sounds weird, because the Travis from the PR is actually green... I never saw this error message before...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135397135
  
    The problem was that the storm tests assume a certain behavior of the `ClusterUtil` which changes as part of the Kafka rework pull request. Should be fixed now, I'll give this another try to merge...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/1037


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135358868
  
    I actually got test errors when trying to merge this:
    https://s3.amazonaws.com/archive.travis-ci.org/jobs/77385135/log.txt


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mjsax <gi...@git.apache.org>.

Github user mjsax commented on a diff in the pull request:

https://github.com/apache/flink/pull/1037#discussion_r37640464

--- Diff: docs/apis/storm_compatibility.md ---
@@ -224,4 +224,12 @@ DataStream<String> rawInput = env.addSource(

This does not work as I want it. The shade plugin overwrites some previously build jars and also installs the example jars to local maven repository. Both is not intended and I was not able to deactivate it... (`<shadedArtifactAttached>true</shadedArtifactAttached>` only keeps a copy of the overwritten jar (with different name).

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-135359067
  
    Consistently failed on all 5 builds with the same errors...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by tillrohrmann <gi...@git.apache.org>.

Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1037#discussion_r37621943
  
    --- Diff: docs/apis/storm_compatibility.md ---
    @@ -224,4 +224,12 @@ DataStream<String> rawInput = env.addSource(
     
     You can find more examples in Maven module `flink-storm-compatibilty-examples`.
     For the different versions of WordCount, see [README.md](https://github.com/apache/flink/tree/master/flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/README.md).
    +To run the examples, you need to assemble a correct jar file.
    +`flink-storm-compatibility-examples-0.10-SNAPSHOT.jar` is **no** valid jar file for job execution (it is only a standard maven artifact).
     
    +There are example jars for embedded Spout and Bolt, namely `flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountSpoutSource.jar` and `flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountBoltTokenizer.jar`, respectively.
    +Compare `pom.xml` to see how both jars are built.
    +Furthermore, there is one example for whole Storm topologies (`flink-storm-compatibility-examples-0.10-SNAPSHOT-WordCountStorm.jar`).
    +This jar is build using `maven-assembly-plugin` because it must contain `defaults.yaml` which cannot be included using `maven-jar-plugin`.
    --- End diff --
    
    This should hopefully do the trick
    
    ```
    <plugin>
    	<artifactId>maven-shade-plugin</artifactId>
    	<groupId>org.apache.maven.plugins</groupId>
    	<version>2.4.1</version>
    	<executions>
    		<execution>
    			<id>StormWordCount</id>
    			<phase>package</phase>
    			<goals>
    				<goal>shade</goal>
    			</goals>
    			<configuration>
    				<outputFile>target/assembly/StormWordCount.jar</outputFile>
    				<minimizeJar>true</minimizeJar>
    				<artifactSet>
    					<includes>
    						<include>org.apache.storm:storm-core</include>
    						<include>org.apache.flink:flink-storm-compatibility-core</include>
    						<include>org.apache.flink:flink-storm-compatibility-examples</include>
    					</includes>
    				</artifactSet>
    				<filters>
    					<filter>
    						<artifact>org.apache.storm:storm-core</artifact>
    						<includes>
    							<include>defaults.yaml</include>
    							<include>backtype/storm/*.class</include>
    							<include>backtype/storm/topology/*.class</include>
    							<include>backtype/storm/spout/*.class</include>
    							<include>backtype/storm/task/*.class</include>
    							<include>backtype/storm/tuple/*.class</include>
    							<include>backtype/storm/generated/*.class</include>
    							<include>backtype/storm/metric/**/*.class</include>
    							<include>backtype/storm/utils/*.class</include>
    							<include>backtype/storm/serialization/*.class</include>
    							<include>org/apache/storm/curator/**/*.class</include>
    							<include>org/apache/thrift7/**/*.class</include>
    							<!-- Storm's recursive dependencies -->
    							<include>org/json/simple/**/*.class</include>
    							<include>org/yaml/snakeyaml/**/*.class</include>
    						</includes>
    					</filter>
    					<filter>
    						<artifact>org.apache.flink:flink-storm-compatibility-examples</artifact>
    						<includes>
    							<include>org/apache/flink/stormcompatibility/wordcount/StormWordCountRemoteBySubmitter.class</include>
    							<include>org/apache/flink/stormcompatibility/wordcount/WordCountTopology.class</include>
    							<include>org/apache/flink/stormcompatibility/wordcount/stormoperators/*.class</include>
    							<include>org/apache/flink/stormcompatibility/util/*.class</include>
    							<include>org/apache/flink/examples/java/wordcount/util/WordCountData.class</include>
    						</includes>
    					</filter>
    					<filter>
    						<artifact>org.apache.flink:flink-storm-compatibility-core</artifact>
    						<includes>
    							<include>org/apache/flink/stormcompatibility/api/*.class</include>
    							<include>org/apache/flink/stormcompatibility/wrappers/*.class</include>
    						</includes>
    					</filter>
    				</filters>
    				<transformers>
    					<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
    						<mainClass>org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter</mainClass>
    					</transformer>
    				</transformers>
    			</configuration>
    		</execution>
    	</executions>
    </plugin>
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by tillrohrmann <gi...@git.apache.org>.

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1037#discussion_r37561072

--- Diff: docs/apis/storm_compatibility.md ---
@@ -224,4 +224,12 @@ DataStream<String> rawInput = env.addSource(

This should also be possible with the maven-shade-plugin, for the sake of completeness.

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by mbalassi <gi...@git.apache.org>.

Github user mbalassi commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1037#discussion_r37661571
  
    --- Diff: flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/wordcount/StormWordCountRemoteBySubmitter.java ---
    @@ -71,7 +71,7 @@ public static void main(final String[] args) throws Exception {
     		// conf.put(Config.NIMBUS_THRIFT_PORT, new Integer(6123));
     
     		// The user jar file must be specified via JVM argument if executed via Java.
    -		// => -Dstorm.jar=target/flink-storm-examples-0.9-SNAPSHOT-WordCountStorm.jar
    +		// => -Dstorm.jar=target/flink-storm-examples-0.10-SNAPSHOT-WordCountStorm.jar
     		// If bin/flink is used, the jar file is detected automatically.
    --- End diff --
    
    I would write `-Dstorm.jar=target/flink-storm-examples-*-WordCountStorm.jar` here and maybe also at the former location if that works. Easier to maintain. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2553] Example Jars not build correctly

Posted by tillrohrmann <gi...@git.apache.org>.

Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/1037#issuecomment-133102088
  
    Only had a minor comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---