You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by chobeat <gi...@git.apache.org> on 2016/09/09 12:32:40 UTC
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
GitHub user chobeat opened a pull request:
https://github.com/apache/flink/pull/2485
[Flink 4599] - Add 'explain()' also to StreamTableEnvironment
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/radicalbit/flink FLINK-4599
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2485.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2485
----
commit 29f0204308a8f871e7b27767efc3020e362218f4
Author: chobeat <si...@gmail.com>
Date: 2016-09-09T09:27:41Z
AST DataStream table explain
commit 593569405bbd2b728f5e9edc82242019a7fbe9be
Author: chobeat <si...@gmail.com>
Date: 2016-09-09T09:31:12Z
Merge remote-tracking branch 'origin/master' into FLINK-4599
commit 7bf135237cb24ed889b92bf5d8026de43d1038fc
Author: f7753 <ma...@gmail.com>
Date: 2016-08-22T13:07:55Z
[FLINK-4436] Unclosed DataOutputBuffer in Utils#setTokensFor()
This closes #2402
commit ba043aaa51401f53c2868927a540ebf7a3493318
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-07-25T13:09:27Z
[FLINK-4257] [gelly] Handle delegating algorithm change of class
Replaces Delegate with NoOpOperator.
This closes #2474
commit 850fd5fec5133f7729bc6a5b2af00cb2decc229b
Author: Till Rohrmann <tr...@apache.org>
Date: 2016-08-31T15:58:09Z
[FLINK-4458] Replace ForkableFlinkMiniCluster by LocalFlinkMiniCluster
Rename _configuration to originalConfiguration
Remove testing classes from main scope in flink-runtime
Previously, the ForkableFlinkMiniCluster which resided in flink-test-utils required
these files to be in the main scope of flink-runtime. With the removal of the
ForkableFlinkMiniCluster, these classes are now no longer needed and can be moved
back to the test scope.
This closes #2450.
commit 7eecc4bd7aa605051cc3c6abc1a87233b8695127
Author: Till Rohrmann <tr...@apache.org>
Date: 2016-09-01T12:41:44Z
[FLINK-4456] Replace Akka specific types by interfaces in Task
Introduce TaskExecutionStateListener for Task
Replace JobManagerGateway in Task by InputSplitProvider and CheckpointNotifier
Replace the TaskManager ActorGateway by TaskManagerConnection in Task
Rename taskmanager.CheckpointNotifier into CheckpointResponder; rename TaskExecutionStateListener.notifyTaskExecutionState into notifyTaskExecutionStateChanged
Remove InputSplitProvider.start; add ClassLoader parameter to InputSplitProvider.getNextInputSplit
Removes the unused class InputSplitIterator.
Update InputSplitProvider JavaDocs
This closes #2456.
commit 45f842eb8ea7da4f0535cd29c345fd45fe3d3815
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-09-02T14:42:30Z
[FLINK-4522] [docs] Gelly link broken in homepage
The Gelly documentation was recently split into multiple pages in
FLINK-4104 but was missing a redirect. This commit updates the Gelly
redirect to point to the old page.
This closes #2464
commit 7c9d1679c8319d560c9032691ad05b723b852f66
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-09-02T15:53:08Z
[FLINK-4571] [gelly] Configurable little parallelism in Gelly drivers
This closes #2475
commit 2ddb4fef89debd7cef740058cc79ddca097879e1
Author: Alexander Pivovarov <ap...@gmail.com>
Date: 2016-09-07T21:11:06Z
[FLINK-4595] Close FileOutputStream in ParameterTool
This closes #2478
commit 2d4c75e688cd143022fc1db2b209c71935003c7d
Author: chobeat <si...@gmail.com>
Date: 2016-09-09T09:32:13Z
Merge branch 'FLINK-4599' of github.com:radicalbit/flink into FLINK-4599
commit f3ba22ba5d2901c25cdbd11689a1ca3cc50935cd
Author: chobeat <si...@gmail.com>
Date: 2016-09-09T12:30:50Z
moved stream explain test resources to main folder
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Posted by apivovarov <gi...@git.apache.org>.
Github user apivovarov commented on a diff in the pull request:
https://github.com/apache/flink/pull/2485#discussion_r78270846
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/StreamTableEnvironment.scala ---
@@ -271,5 +271,21 @@ abstract class StreamTableEnvironment(
}
}
+ /*
+ * Returns the AST of the specified Table API and SQL queries and the execution plan to compute
+ * the result of the given [[Table]].
+ *
+ * @param table The table for which the AST and execution plan will be returned.
+ * @param extended Flag to include detailed optimizer estimates.
+ */
+ def explain(table: Table): String = {
+
+ val ast = RelOptUtil.toString(table.getRelNode)
+
+ s"== Abstract Syntax Tree ==" +
+ System.lineSeparator +
+ s"$ast"
--- End diff --
No need for s on line 285
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Posted by apivovarov <gi...@git.apache.org>.
Github user apivovarov commented on a diff in the pull request:
https://github.com/apache/flink/pull/2485#discussion_r78270898
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/scala/stream/ExplainStreamTest.scala ---
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.stream
+
+import org.apache.flink.api.scala._
+import org.apache.flink.api.scala.table._
+import org.apache.flink.api.table.TableEnvironment
+import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
+import org.apache.flink.streaming.util.StreamingMultipleProgramsTestBase
+import org.junit.Assert.assertEquals
+import org.junit._
+
+class ExplainStreamTest
+ extends StreamingMultipleProgramsTestBase {
+
+ val testFilePath = ExplainStreamTest.this.getClass.getResource("/").getFile
+
+ @Test
+ def testFilter() : Unit = {
+ val env = StreamExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env)
+
+ val table = env.fromElements((1, "hello"))
+ .toTable(tEnv, 'a, 'b)
+ .filter("a % 2 = 0")
+
+ val result = tEnv.explain(table).replaceAll("\\r\\n", "\n")
+ val source = scala.io.Source.fromFile(testFilePath +
+ "../../src/test/scala/resources/testFilterStream0.out").mkString.replaceAll("\\r\\n", "\n")
+ assertEquals(result, source)
+ }
+
+ @Test
+ def testUnion() : Unit = {
--- End diff --
no need for space after () on line 35 and 50
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2485: [Flink 4599] - Add 'explain()' also to StreamTableEnviron...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/2485
Could you rebase your PR on the current master and then do a force push? There are a lot of commits that should not appear here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Posted by apivovarov <gi...@git.apache.org>.
Github user apivovarov commented on a diff in the pull request:
https://github.com/apache/flink/pull/2485#discussion_r78270796
--- Diff: docs/dev/table_api.md ---
@@ -2457,3 +2457,27 @@ The Table API provides a configuration (the so-called `TableConfig`) to modify r
By default, the Table API supports `null` values. Null handling can be disabled to improve preformance by setting the `nullCheck` property in the `TableConfig` to `false`.
{% top %}
+
+Explaining a Table
+----
+The Table API provides a mechanism to describe the graph of operations that leads to the resulting output. This is done through the `TableEnvironment#explain(table)` method. It returns a string describing two graphs: the Abstract Syntax Tree of the relational algebra query and the Flink's Execution Plan of the equivalent Flink's Job.
+
+Table `explain` is supported for both `BatchTableEnvironment` and `StreamTableEnvironment`. Currently `StreamTableEnvironment` doesn't support the explanation of the Execution Plan.
+
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+ val env = StreamExecutionEnvironment.getExecutionEnvironment
+ val tEnv = TableEnvironment.getTableEnvironment(env)
+
+ val table1 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word)
+ val table2 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word)
+ val table = table1.unionAll(table2)
+
+ val explanation:String = tEnv.explain(table)
--- End diff --
put space before String
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2485
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2485: [Flink 4599] - Add 'explain()' also to StreamTable...
Posted by apivovarov <gi...@git.apache.org>.
Github user apivovarov commented on a diff in the pull request:
https://github.com/apache/flink/pull/2485#discussion_r78270826
--- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/StreamTableEnvironment.scala ---
@@ -271,5 +271,21 @@ abstract class StreamTableEnvironment(
}
}
+ /*
+ * Returns the AST of the specified Table API and SQL queries and the execution plan to compute
+ * the result of the given [[Table]].
+ *
+ * @param table The table for which the AST and execution plan will be returned.
+ * @param extended Flag to include detailed optimizer estimates.
+ */
+ def explain(table: Table): String = {
+
+ val ast = RelOptUtil.toString(table.getRelNode)
+
+ s"== Abstract Syntax Tree ==" +
+ System.lineSeparator +
+ s"$ast"
--- End diff --
Maybe this?
```
s"== Abstract Syntax Tree ==${System.lineSeparator}$ast"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2485: [Flink 4599] - Add 'explain()' also to StreamTableEnviron...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/2485
Thanks @chobeat. I fixed minor formatting issues and extended the documentation a little bit. Will merge...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2485: [Flink 4599] - Add 'explain()' also to StreamTableEnviron...
Posted by chobeat <gi...@git.apache.org>.
Github user chobeat commented on the issue:
https://github.com/apache/flink/pull/2485
Done, sorry.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---