You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by mbalassi <gi...@git.apache.org> on 2014/11/12 23:57:36 UTC
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
GitHub user mbalassi opened a pull request:
https://github.com/apache/incubator-flink/pull/199
[scala] Self-contained build for scala examples
The scala examples lack self-contained build, because the default data is located in the flink-java-examples jar. Here is a proposed solution for that.
@aljoscha, @rmetzger what do you think?
Cheers,
Marton
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mbalassi/incubator-flink scala-examples
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #199
----
commit 42184b2ea27481524b0179e375d58ff4f184087c
Author: mbalassi <mb...@apache.org>
Date: 2014-11-12T22:47:40Z
[scala] Self-contained build for scala examples
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62940685
Tested and merged in 9ff2e5b32d0e23f0c9ee817ea1a86a71b2c4698d.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62901286
OK, sorry for the confusion. +1
I think that @StephanEwen also thought that we were talking about adding it to the examples directory.
If he is also OK with this, I can merge it later today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-flink/pull/199
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/199#discussion_r20279295
--- Diff: flink-examples/flink-scala-examples/pom.xml ---
@@ -170,164 +170,283 @@ under the License.
</configuration>
</plugin>
+ <!-- get default data from flink-java-examples package -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>2.9</version>
+ <executions>
+ <execution>
+ <id>unpack</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>unpack</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <artifactItem>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-java-examples</artifactId>
+ <version>${project.version}</version>
+ <type>jar</type>
+ <overWrite>false</overWrite>
+ <outputDirectory>${project.build.directory}/classes</outputDirectory>
+ <includes>**/util/*Data*.class</includes>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
- <!-- These examples are currently not self-contained
-
+
+ <!-- KMeans -->
<execution>
<id>KMeans</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
-
+
<configuration>
<classifier>KMeans</classifier>
+
<archive>
<manifestEntries>
- <program-class>org.apache.flink.examples.scala.datamining.KMeans</program-class>
+ <program-class>org.apache.flink.examples.scala.clustering.KMeans</program-class>
</manifestEntries>
</archive>
-
+
<includes>
- <include>**/datamining/KMeans*.class</include>
+ <include>**/scala/clustering/KMeans.class</include>
+ <include>**/scala/clustering/KMeans$*.class</include>
+ <include>**/java/clustering/util/KMeansDataGenerator.class</include>
+ <include>**/java/clustering/util/KMeansData.class</include>
</includes>
</configuration>
</execution>
-
+
+ <!-- Transitive Closure -->
<execution>
- <id>ComputeEdgeDegrees</id>
+ <id>TransitiveClosure</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
-
<configuration>
- <classifier>ComputeEdgeDegrees</classifier>
-
+ <classifier>TransitiveClosure</classifier>
+
<archive>
<manifestEntries>
- <program-class>org.apache.flink.examples.scala.graph.ComputeEdgeDegrees</program-class>
+ <program-class>org.apache.flink.examples.java.scala.TransitiveClosureNaive</program-class>
--- End diff --
`java.scala` is probably not right.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62902648
No problem, this just means that I should have been more clear with the explanation of the PR.
Thanks for picking up the merge @uce.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62869820
It was actually a conscious choice to have all examples once. It does not really add value to have the examples twice. When all you have as a jar with bytecode, and both versions behave the same, why would you care what language the byte code was compiled from?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62870494
Just to clarify I do not want to add these to the examples directory of the Flink distribution. I just would like to have them sitting in the flink-scala-examples/target if someone was interested.
I can accept that to be invaluable if you say so, but at least adding the data classes should be there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62862753
What does this commit change exactly?
Are there jar files in the examples/ directory for the Scala examples?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-flink pull request: [scala] Self-contained
build for sca...
Posted by Ufuk Celebi <uc...@apache.org>.
I think it makes very much sense to have this setup with the data classes.
+1 on that.
My question was only when *users* might want to run the Scala examples
instead of the Java examples since they do the same thing.
On Thu, Nov 13, 2014 at 11:14 AM, mbalassi <gi...@git.apache.org> wrote:
> Github user mbalassi commented on the pull request:
>
>
> https://github.com/apache/incubator-flink/pull/199#issuecomment-62869095
>
> The commit does two things:
> * Adds the default data classes to the examples package from the
> flink-java-examples packages
> * Also creates separate jars for the examples
>
> I suppose both makes it more convenient for the users to try out the
> programs on the cluster. Currently to see a scala example performing on the
> cluster you would have to add the flink-java-examples (as it is not in the
> lib of the distro) to the classpath and of course also provide the desired
> main class.
>
> I would not say that this is difficult, just the option I'm providing
> is more user friendly.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
Re: [GitHub] incubator-flink pull request: [scala] Self-contained
build for sca...
Posted by Ufuk Celebi <uc...@apache.org>.
I think it makes very much sense to have this setup with the data classes.
+1 on that.
My question was only when *users* might want to run the Scala examples
instead of the Java examples since they do the same thing.
On Thu, Nov 13, 2014 at 11:14 AM, mbalassi <gi...@git.apache.org> wrote:
> Github user mbalassi commented on the pull request:
>
>
> https://github.com/apache/incubator-flink/pull/199#issuecomment-62869095
>
> The commit does two things:
> * Adds the default data classes to the examples package from the
> flink-java-examples packages
> * Also creates separate jars for the examples
>
> I suppose both makes it more convenient for the users to try out the
> programs on the cluster. Currently to see a scala example performing on the
> cluster you would have to add the flink-java-examples (as it is not in the
> lib of the distro) to the classpath and of course also provide the desired
> main class.
>
> I would not say that this is difficult, just the option I'm providing
> is more user friendly.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62869095
The commit does two things:
* Adds the default data classes to the examples package from the flink-java-examples packages
* Also creates separate jars for the examples
I suppose both makes it more convenient for the users to try out the programs on the cluster. Currently to see a scala example performing on the cluster you would have to add the flink-java-examples (as it is not in the lib of the distro) to the classpath and of course also provide the desired main class.
I would not say that this is difficult, just the option I'm providing is more user friendly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62902065
Agreed, building them to target in scala-examples is good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-flink pull request: [scala] Self-contained
build for sca...
Posted by Ufuk Celebi <uc...@apache.org>.
I think separate Scala JARs don't add much value since we have the examples
already packaged. As source they make very much sense, but is there a
difference for users when trying out the system?
On Thu, Nov 13, 2014 at 10:21 AM, aljoscha <gi...@git.apache.org> wrote:
> Github user aljoscha commented on the pull request:
>
>
> https://github.com/apache/incubator-flink/pull/199#issuecomment-62862942
>
> Right now there are no jar files since the scala examples depend on
> the example data of the java examples. I think this is a good solution, if
> we want to have the Scala examples as jars.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
Re: [GitHub] incubator-flink pull request: [scala] Self-contained
build for sca...
Posted by Ufuk Celebi <uc...@apache.org>.
I think separate Scala JARs don't add much value since we have the examples
already packaged. As source they make very much sense, but is there a
difference for users when trying out the system?
On Thu, Nov 13, 2014 at 10:21 AM, aljoscha <gi...@git.apache.org> wrote:
> Github user aljoscha commented on the pull request:
>
>
> https://github.com/apache/incubator-flink/pull/199#issuecomment-62862942
>
> Right now there are no jar files since the scala examples depend on
> the example data of the java examples. I think this is a good solution, if
> we want to have the Scala examples as jars.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] incubator-flink pull request: [scala] Self-contained build for sca...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/incubator-flink/pull/199#issuecomment-62862942
Right now there are no jar files since the scala examples depend on the example data of the java examples. I think this is a good solution, if we want to have the Scala examples as jars.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---