You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by shijinkui <gi...@git.apache.org> on 2016/09/02 07:01:19 UTC
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
GitHub user shijinkui opened a pull request:
https://github.com/apache/flink/pull/2460
[FLINK-4562] table examples make an divided module in flink-examples
only move table examle code to a divided module in flink-examples.
Table API module should't contain example code.
[issue](https://issues.apache.org/jira/browse/FLINK-4562)
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-4562] table examples make an divided module in flink-examples")
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shijinkui/flink table-example-module
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2460.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2460
----
commit 4fb919b7bc31a269bfa82e32ae67fb43065ba6ac
Author: shijinkui <sh...@huawei.com>
Date: 2016-09-02T06:50:11Z
[FLINK-4562] table examples make an divided module in flink-examples
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2460
I agree. +1 for moving the examples.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
ping @wuchong @fhueske @twalthr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2460
Thanks, @shijinkui
I think this PR is good to merge. I was waiting for you to add the examples jar file to the `./opt` folder, but we can also do this as a follow up task.
I'll rename the resulting file to `flink-table-examples-with-dependencies.jar` and merge this PR.
Thanks,
Fabian
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2460
Hi @shijinkui, yes that sounds good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
@fhueske Thanks : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
> Hi @shijinkui, yes that sounds good to me.
Thanks. When will we merge this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2460#discussion_r107804086
--- Diff: flink-examples/flink-examples-table/pom.xml ---
@@ -0,0 +1,108 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-examples_2.10</artifactId>
+ <version>1.3-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <name>flink-examples-table</name>
+ <artifactId>flink-examples-table_2.10</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table_2.10</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>net.alchim31.maven</groupId>
+ <artifactId>scala-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>scala-compile-first</id>
+ <phase>process-resources</phase>
+ <goals>
+ <goal>add-source</goal>
+ <goal>compile</goal>
+ </goals>
+ </execution>
+ <execution>
+ <id>scala-test-compile</id>
+ <phase>process-test-resources</phase>
+ <goals>
+ <goal>testCompile</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>fat-jar-table-examples</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadeTestJar>false</shadeTestJar>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <createDependencyReducedPom>false</createDependencyReducedPom>
+ <finalName>TableExamples</finalName>
--- End diff --
I think we should keep the name of the generated artifact as `flink-table-examples_2.10`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2460
Thanks for the reminder @shijinkui!
I'll put this one on my list for next week.
Thanks, Fabian
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
@wuchong @StephanEwen can you review this pull request.
Generating independent example runnable jar will increase the size of flink distribution.
This table example jar is 13Mb or so.
We can make a agreement of example here:
1. all the example code moved in flink-examples module, as a sub-module. Yes or No?
2. optionally generate runnable example jar in flink distribution. Yes or No?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
fixed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
> I agree with @wuchong , that we should follow the example of Gelly and add the flink-table-examples JAR file to the opt folder.
@fhueske Thanks for your review.
IMO, directory style below is reasonable:
* `examples` directory only contain example jars
* `opt` directory only contain optional library jars
* `lib` directory only contain library jar that must be load in runtime
The `opt` directory is noisy, that contains lib jar and example jar:
```
flink-cep-scala_2.11-1.3.0.jar
flink-gelly_2.11-1.3.0.jar
flink-metrics-statsd-1.3.0.jar
flink-cep_2.11-1.3.0.jar
flink-metrics-dropwizard-1.3.0.jar
flink-ml_2.11-1.3.0.jar
flink-gelly-examples_2.11-1.3.0.jar
flink-metrics-ganglia-1.3.0.jar
flink-gelly-scala_2.11-1.3.0.jar
flink-metrics-graphite-1.3.0.jar
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/flink/pull/2460#discussion_r80955533
--- Diff: flink-examples/flink-examples-table/pom.xml ---
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-examples_2.10</artifactId>
+ <version>1.2-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>flink-examples-table_2.10</artifactId>
+ <name>flink-examples-table</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table_2.10</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
--- End diff --
I lost <packaging>jar</packaging>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2460
The repackaging is good.
Unfortunately this breaks the Table API tests, because they use one of the examples.
The Table API IT cases probably need to use their own programs, rather than relying on the examples.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on the issue:
https://github.com/apache/flink/pull/2460
Could we move the Scala examples to a separate package , such as: `org.apache.flink.table.scala.examples`, to avoid potential class conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/flink/pull/2460#discussion_r80955413
--- Diff: flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/StreamSQLExample.scala ---
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.flink.examples.scala
+package org.apache.flink.table.examples
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
> The repackaging is good.
> Unfortunately this breaks the Table API tests, because they use one of the examples.
>
> The Table API IT cases probably need to use their own programs, rather than relying on the examples.
Table test dependency the `WC` case class of WordCountExample. I have divided them.
the flink-examples-table module have support building the fat jar like `flink-examples-streaming` and `flink-examples-batch`. IMO, all the example module should move to the `flink-examples` module as a sub-module :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2460
I think this is actually a good idea. The Table API module should not contain the examples.
@fhueske @twalthr What do you think about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2460#discussion_r107792137
--- Diff: flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamTableExample.scala ---
@@ -18,18 +18,18 @@
package org.apache.flink.table.examples.scala
import org.apache.flink.api.scala._
-import org.apache.flink.table.api.scala._
import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment}
import org.apache.flink.table.api.TableEnvironment
+import org.apache.flink.table.api.scala._
/**
- * Simple example for demonstrating the use of Table API on a Stream Table.
- *
- * This example shows how to:
- * - Convert DataStreams to Tables
- * - Apply union, select, and filter operations
- *
- */
+ * Simple example for demonstrating the use of Table API on a Stream Table.
--- End diff --
Can you fix the Scaladoc comment style of the other Scala classes as well? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2460#discussion_r80916077
--- Diff: flink-examples/flink-examples-table/pom.xml ---
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-examples_2.10</artifactId>
+ <version>1.2-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>flink-examples-table_2.10</artifactId>
+ <name>flink-examples-table</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table_2.10</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
--- End diff --
My IDE (IntelliJ) does not recognize `src/scala` as source folder. It might be necessary to add additional Maven configuration. I would use `flink-gelly-examples` as a reference for the plugin configuration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on the issue:
https://github.com/apache/flink/pull/2460
Hi @shijinkui , sorry for the delay, the change looks very good to me. Regarding to your questions,
>1. all the example code moved in flink-examples module, as a sub-module. Yes or No?
Yes, it's fine to me.
>2. optionally generate runnable example jar in flink distribution. Yes or No?
I think we can generate the runnable table example jar in the distribution. And I think we can put it under
`/examples/`. But I find that the gelly example is put into `/opt/`. Maybe we can start a discussion in mailing list to keep consistent with gelly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/2460#discussion_r80910964
--- Diff: flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/StreamSQLExample.scala ---
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.flink.examples.scala
+package org.apache.flink.table.examples
--- End diff --
This should be called `org.apache.flink.table.examples.scala` to be consistent with `org.apache.flink.table.examples.java`. IMHO the language should be right after the `table`: `org.apache.flink.table.java.examples`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2460
Merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by shijinkui <gi...@git.apache.org>.
Github user shijinkui commented on the issue:
https://github.com/apache/flink/pull/2460
@wuchong you are right. Different package of scala/java can avoid same class conflict
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2460: [FLINK-4562] table examples make an divided module...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2460
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2460: [FLINK-4562] table examples make an divided module in fli...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/2460
+1
I will shepherd this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---