You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by sunjincheng121 <gi...@git.apache.org> on 2017/05/19 09:43:57 UTC
[GitHub] flink pull request #3943: [FLINK-6617][table] Improve JAVA and SCALA logical...
GitHub user sunjincheng121 opened a pull request:
https://github.com/apache/flink/pull/3943
[FLINK-6617][table] Improve JAVA and SCALA logical plans consistent test
In this PR. I had make some changes as follows:
1. Add all the tableAPI AVA and SCALA logical plans consistent test.
2. Add `validation` folder for steam which keep consistent with batch.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-6617][table] Improve JAVA and SCALA logical plans consistent test")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sunjincheng121/flink FLINK-6617-PR
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3943.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3943
----
commit 01d7398238b49eae45138ed7318969c61e52424a
Author: sunjincheng121 <su...@gmail.com>
Date: 2017-05-18T05:02:24Z
[FLINK-6617][table] Improve JAVA and SCALA logical plans consistent test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Hi @wuchong Thanks for your approval.
In addition to refactoring the test structure, I think that in addition to ITCast other tests should be unified use of `batchTestUtil` and` streamTestUtil`. What do you think? @wuchong @twalthr @fhueske
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/3943
@wuchong @sunjincheng121 @fhueske
I tried to simplify the package structure a little bit. I based my changes on top of this PRs commit.
My goals:
- splitting a class like (`ExpressionReductionTest`) in 6 classes is not very helpful
- tests are named like the operator or feature that they are testing e.g. `Calc`, `Distinct` etc. (we don't need `CastingITCase`, `CalcITCase`, `ScalarFunctionITCase`; `CalcITCase` covers everything)
- all tests have the same name everywhere (`OverWindowTest`, `OverWindowValidationTest`, `OverWindowITCase`, `OverWindowStringExpressionTest`)
- no packages with one class
- `org.apache.flink.table.api` contains all tests that test the translation from batch/stream/table/sql APIs
- `org.apache.flink.table.plan` contains all tests that modify the plan
- `org.apache.flink.table.runtime` contains all ITCases and runtime relevant tests
- all other package contain unit tests for the classes of the same package
You can find my branch here: https://github.com/apache/flink/compare/master...twalthr:FLINK-6617
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3943: [FLINK-6617][table] Improve JAVA and SCALA logical...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/3943#discussion_r117929561
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/table/stringexpr/UnionStringExpressionTest.scala ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.api.scala.stream.table.stringexpr
+
+import org.apache.flink.api.scala._
+import org.apache.flink.table.api.scala._
+import org.apache.flink.table.utils.TableTestBase
+import org.junit.Test
+
+class UnionStringExpressionTest extends TableTestBase {
+ @Test
--- End diff --
Add an empty line here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3943: [FLINK-6617][table] Improve JAVA and SCALA logical...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/3943#discussion_r117928792
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/table/validation/GroupWindowAggregationsTest.scala ---
@@ -16,19 +16,19 @@
* limitations under the License.
*/
-package org.apache.flink.table.api.scala.stream.table
+package org.apache.flink.table.api.scala.stream.table.validation
--- End diff --
I think tests in `validation` packages should all test for a `ValidationException`. Can you split this class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Hi @twalthr Thanks a log for your reviewing. I completely agree with your suggestions. I petty want refactoring the test structure. I have updated the PR. according your comments.
Thanks,
SunJincheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Rebase code according master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3943: [FLINK-6617][table] Improve JAVA and SCALA logical...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3943
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
May be I need split this PR into a couple of small PRs. What do you think? or have any other suggestions @fhueske @wuchong @twalthr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3943: [FLINK-6617][table] Improve JAVA and SCALA logical...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/3943#discussion_r117929380
--- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/table/utils/TableTestBase.scala ---
@@ -120,7 +125,8 @@ object TableTestUtil {
}
case class BatchTableTestUtil() extends TableTestUtil {
-
+ val javaEnv = mock(classOf[JExecutionEnvironment])
+ val javaTableEnv = TableEnvironment.getTableEnvironment(javaEnv)
--- End diff --
Maybe we can rename the Scala environment to `tableEnv` to be consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/3943
Thanks for the update @sunjincheng121. I will go over all changes and try to merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Thanks @twalthr ! I have rebased the code and updated the PR.
Thanks,
SunJincheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/3943
@sunjincheng121 the structure looks very good. I would volunteer to go over the changes as a whole and merge this. If @fhueske @wuchong are fine with this? Can you rebase the PR a last time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on the issue:
https://github.com/apache/flink/pull/3943
Thanks @twalthr , I'm fine with this ,a great +1!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/3943
I think that is a very good idea. A review of a +6k -4k LOC PR takes a lot of time and has a good chance to be interrupted by other issues. Several smaller PRs would be much easier to review and merge.
It would be great if you could do that @sunjincheng121.
Thank you very much!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by wuchong <gi...@git.apache.org>.
Github user wuchong commented on the issue:
https://github.com/apache/flink/pull/3943
Hi @sunjincheng121 , thanks for your great work. I like your proposal. The test structure refactoring looks good to me. I think it's good to put all the IT cases in one place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/3943
Hi @twalthr and @sunjincheng121, I looked over the package structure and naming of the tests and this looks good to me.
+1 to merge from my side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Update again. Because the new merged code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Rebase code...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
@twalthr I have rebased code against master.
Best,
SunJincheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
@twalthr I have rebase the PR. I appreciated if you can review it. And i'll quick updated it when you left comments.
Thanks,
SunJincheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/3943
Merging...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
**I once again seriously consider the Refactoring work, according to the following principles updated the PR.**
**First, the test type and purpose(4 types):**
* Unit Test: to ensure the correctness of the most granular features
* StringExpression Test: Verify the consistency of java and scalaAPI
* Validation Test: Check the API exception information
* Integration Test: Integration testing, verifying runtime and end-to-end correctness
**Second, the directory structure(folders) and purpose:**
The TEST directory contains all the paths (folders) of the SRC directory - the functional tests included three type test case (**Unit Test**, **StringExpression Test**,**Validation Test**)
**Note:** runtime directory contains both runtime (**Unit Test**, **StringExpression Test**,**Validation Test**), and All table API/SQL **Integration Test**.
**Third, the file name naming convention:**
* Unit Test -> functionName + **Test**
* StringExpression Test -> functionName + **StringExpressionTest**
* Validation Test -> functionName + **ValidationTest**
* Integration Test -> FunctionName + **ITCase**
**After the change of the directory structure, as shown below:**
@twalthr @fhueske @wuchong The change is big, so I need your help to ensure that the direction of change is correct.
Best,
SunJincheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
@twalthr Extremely grateful If you can review and merge this PR after I rebase the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/3943
Hi @twalthr @fhueske thanks a lot for your review.
@twalthr your improve changes are make the structure of the package is clearer which make sense to me.
+1 to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3943: [FLINK-6617][table] Improve JAVA and SCALA logical plans ...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/3943
@twalthr, sure! Thanks for taking care of this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---