You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by jatin9896 <gi...@git.apache.org> on 2018/02/15 15:09:33 UTC
[GitHub] carbondata pull request #1981: [Pre-Agg Test] Added SDV TestCase of preaggre...
GitHub user jatin9896 opened a pull request:
https://github.com/apache/carbondata/pull/1981
[Pre-Agg Test] Added SDV TestCase of preaggregate
1. Added test cases for pre-aggregate create, load,
2. Added test cases for time series in pre-aggregate.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed? No
- [ ] Any backward compatibility impacted? No
- [ ] Document update required? No
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jatin9896/incubator-carbondata sdvTestCasePreAggregate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1981.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1981
----
commit 092445b75423f327d503b0210789d3de7525642d
Author: Jatin <ja...@...>
Date: 2018-02-15T14:55:20Z
Added SDV TestCase of preaggregate
----
---
[GitHub] carbondata pull request #1981: [Pre-Agg Test] Added SDV TestCase of preaggre...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1981#discussion_r175446224
--- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/PreAggregateTestCase.scala ---
@@ -0,0 +1,312 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.cluster.sdv.generated
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util.{Include, QueryTest}
+import org.junit.Assert
+import org.scalatest.BeforeAndAfterEach
+import org.scalatest.Matchers._
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+
+class PreAggregateTestCase extends QueryTest with BeforeAndAfterEach {
+ val csvPath = s"$resourcesPath/source.csv"
+
+ override def beforeEach: Unit = {
+ sql("drop table if exists PreAggMain")
+ sql("drop table if exists AggMain")
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
+ sql("CREATE TABLE PreAggMain (id Int, date date, country string, phonetype string, " +
+ "serialname String,salary int ) STORED BY 'org.apache.carbondata.format' " +
+ "tblproperties('dictionary_include'='country')")
+ sql("CREATE TABLE AggMain (id Int, date date, country string, phonetype string, " +
+ "serialname String,salary int ) STORED BY 'org.apache.carbondata.format'" +
+ "tblproperties('dictionary_include'='country')")
+ sql("create datamap PreAggSum on table PreAggMain using 'preaggregate' as " +
+ "select country,sum(salary) as sum from PreAggMain group by country")
+ sql("create datamap PreAggAvg on table PreAggMain using 'preaggregate' as " +
+ "select country,avg(salary) as avg from PreAggMain group by country")
+ sql("create datamap PreAggCount on table PreAggMain using 'preaggregate' as " +
+ "select country,count(salary) as count from PreAggMain group by country")
+ sql("create datamap PreAggMin on table PreAggMain using 'preaggregate' as " +
+ "select country,min(salary) as min from PreAggMain group by country")
+ sql("create datamap PreAggMax on table PreAggMain using 'preaggregate' as " +
+ "select country,max(salary) as max from PreAggMain group by country")
+ }
+
+ //test to check the table and datamap creation
+ test("PreAggregateTestCase_TC001", Include) {
+ checkExistence(sql("Describe formatted PreAggMain"), true)
+ Assert.assertEquals(sql("show datamap on table PreAggMain").count(), 5)
+ }
+
+ //test to check preaggregate table with dictionary
+ test("PreAggregateTestCase_TC002", Include) {
+ checkExistence(sql("Describe formatted PreAggMain_PreAggSum"), true, "DICTIONARY")
+ }
+
+ //test for exception in datamap having wrong column in group by
+ test("PreAggregateTestCase_TC003", Include) {
+ intercept[Exception] {
+ sql("create datamap testDatamap on table PreAggMain using 'preaggregate' as " +
+ "select name,sum(salary) as sum from PreAggMain group by country")
+ }
+ }
+ // test for the exception in datamap having nested aggregate
+ test("PreAggregateTestCase_TC004", Include) {
+ intercept[Exception] {
+ sql("create testDatamap PreAggSum on table PreAggMain using 'preaggregate' as " +
+ "select country,sum(distinct(salary)) as sum from PreAggMain group by country")
+ }
+ }
+
+ // test for the exception in datamap having nested aggregate
+ test("PreAggregateTestCase_TC005", Include) {
--- End diff --
no need for this scenario. Already covered in PreAggregateTestCase_TC004
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by jatin9896 <gi...@git.apache.org>.
Github user jatin9896 commented on the issue:
https://github.com/apache/carbondata/pull/1981
retest sdv please
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3801/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by jatin9896 <gi...@git.apache.org>.
Github user jatin9896 commented on the issue:
https://github.com/apache/carbondata/pull/1981
retest sdv please
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4277/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2556/
---
[GitHub] carbondata pull request #1981: [Pre-Agg Test] Added SDV TestCase of preaggre...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1981
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/1981
Please remove the test cases which are already covered in Functional tests(eg. Negative scenarios, compaction).
---
[GitHub] carbondata pull request #1981: [Pre-Agg Test] Added SDV TestCase of preaggre...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1981#discussion_r174716702
--- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/PreAggregateTestCase.scala ---
@@ -0,0 +1,312 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.cluster.sdv.generated
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util.{Include, QueryTest}
+import org.junit.Assert
+import org.scalatest.BeforeAndAfterEach
+import org.scalatest.Matchers._
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+
+class PreAggregateTestCase extends QueryTest with BeforeAndAfterEach {
+ val csvPath = s"$resourcesPath/source.csv"
+
+ override def beforeEach: Unit = {
+ sql("drop table if exists PreAggMain")
+ sql("drop table if exists AggMain")
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
+ sql("CREATE TABLE PreAggMain (id Int, date date, country string, phonetype string, " +
+ "serialname String,salary int ) STORED BY 'org.apache.carbondata.format' " +
+ "tblproperties('dictionary_include'='country')")
+ sql("CREATE TABLE AggMain (id Int, date date, country string, phonetype string, " +
+ "serialname String,salary int ) STORED BY 'org.apache.carbondata.format'" +
+ "tblproperties('dictionary_include'='country')")
+ sql("create datamap PreAggSum on table PreAggMain using 'preaggregate' as " +
+ "select country,sum(salary) as sum from PreAggMain group by country")
+ sql("create datamap PreAggAvg on table PreAggMain using 'preaggregate' as " +
+ "select country,avg(salary) as avg from PreAggMain group by country")
+ sql("create datamap PreAggCount on table PreAggMain using 'preaggregate' as " +
+ "select country,count(salary) as count from PreAggMain group by country")
+ sql("create datamap PreAggMin on table PreAggMain using 'preaggregate' as " +
+ "select country,min(salary) as min from PreAggMain group by country")
+ sql("create datamap PreAggMax on table PreAggMain using 'preaggregate' as " +
+ "select country,max(salary) as max from PreAggMain group by country")
+ }
+
+ //test to check the table and datamap creation
+ test("PreAggregateTestCase_TC001", Include) {
+ checkExistence(sql("Describe formatted PreAggMain"), true)
--- End diff --
add some criteria to checkExistence on
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3732/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3797/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3678/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/1981
LGTM
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2731/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3604/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3590/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3976/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3033/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3751/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2553/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3587/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2511/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3814/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4432/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3578/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1981
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3953/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3199/
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/1981
retest sdv please
---
[GitHub] carbondata issue #1981: [Pre-Agg Test] Added SDV TestCase of preaggregate
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1981
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2570/
---