You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2018/01/24 09:38:04 UTC
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/1856
[CARBONDATA-2073][CARBONDATA-1516][Tests] Add test cases for timeseries pre-aggregate table
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
No
- [ ] Testing done
Add test cases for timeseries pre-aggregate table
in package org.apache.carbondata.integration.spark.testsuite.timeseries
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
refer [CARBONDATA-1516]
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata PreAggregateTestForTimeSeries
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1856.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1856
----
commit 66d7d0d97a706032459f2dd6b01c95053f437f79
Author: xubo245 <60...@...>
Date: 2018-01-24T09:35:24Z
[CARBONDATA-2073][CARBONDATA-1516][Tests] Add test cases for timeseries pre-aggregate table
----
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3376/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3544/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174060918
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesMatchStrategySuite.scala ---
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesMatchStrategySuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+ var timestampFormat: String = _
--- End diff --
no need for this variable. simply set CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT as the timestamp format is after all.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4085/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
Any one can help to review it? Please. I rebased this PR more than 8 times... @ravipesala @jackylk @QiangCai @chenliang613 @sraghunandan @kumarvishal09
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3103/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3339/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2190/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2724/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2944/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3153/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3723/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3379/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3638/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2234/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174066548
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala ---
@@ -33,16 +33,15 @@ class TestSortColumns extends QueryTest with BeforeAndAfterAll {
CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "dd-MM-yyyy")
SparkUtil4Test.createTaskMockUp(sqlContext)
- dropTable
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "dd-MM-yyyy")
+ dropTestTables
--- End diff --
not required
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3238/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2961/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3092/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r165051330
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesMatchStrategySuite.scala ---
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesMatchStrategySuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+ override def beforeAll: Unit = {
+ SparkUtil4Test.createTaskMockUp(sqlContext)
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+
+ }
+
+ override protected def beforeEach(): Unit = {
+ sql("drop table if exists mainTable")
+ sql(
+ """
+ | CREATE TABLE mainTable(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
+ }
+
+ val timeSeries = "preaggregate"
+
+ test("test timeseries match 1: select small size when there are >1 timeseries match") {
+
+ dropDataMaps("maintable", "agg1_minute", "agg2_minute")
+ sql(
+ s"""
+ | create datamap agg1 on table mainTable
--- End diff --
ok,done
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2198/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2240/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174063930
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesUnsupportedSuite.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesUnsupportedSuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
--- End diff --
I don't think this Unsupported suite is required. Already have test cases for each scenario. No need to test the same for timeseries. It will increase the build time.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest sdv please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3194/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3084/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@kunal642 Please check
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3203/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2259/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2004/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3842/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174060207
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -354,21 +390,137 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
checkExistence(sql("DESC FORMATTED mainTable_agg1"), true, "maintable_age_sum")
}
+ test("test timeseries create table 32: should support if not exists, create when same table not exists") {
+ sql("DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ sql(
+ s"""
+ |CREATE DATAMAP if not exists agg1_year ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'YEAR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_year")
+ checkExistence(sql("DESC FORMATTED mainTable_agg1_year"), true, "maintable_age_sum")
+ }
+
test("test timeseries create table 20: don't support 'create datamap if exists'") {
val e: Exception = intercept[AnalysisException] {
sql(
s"""CREATE DATAMAP IF EXISTS agg2 ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'MONTH_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ | USING '$timeSeries'
--- End diff --
revert this change. it is unncessary
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174058314
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -241,12 +311,12 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
assert(e.getMessage.equals("Only one granularity level can be defined"))
}
- test("test timeseries create table 14: Only one granularity level can be defined 2") {
- sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ test("test timeseries create table 17: Only one granularity level can be defined 2") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
--- End diff --
why change just the name of the table??
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4206/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1878/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1856
LGTM
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3480/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
rebase again
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174406193
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesUnsupportedSuite.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesUnsupportedSuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
--- End diff --
you can find these in preaggregate test cases. No need check the same for timeseries because the base code is the same
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3386/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@ravipesala @jackylk @sraghunandan @QiangCai Please review it.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2164/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3886/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3110/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3675/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3510/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r164660048
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesMatchStrategySuite.scala ---
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesMatchStrategySuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+ override def beforeAll: Unit = {
+ SparkUtil4Test.createTaskMockUp(sqlContext)
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+
+ }
+
+ override protected def beforeEach(): Unit = {
+ sql("drop table if exists mainTable")
+ sql(
+ """
+ | CREATE TABLE mainTable(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
+ }
+
+ val timeSeries = "preaggregate"
+
+ test("test timeseries match 1: select small size when there are >1 timeseries match") {
+
+ dropDataMaps("maintable", "agg1_minute", "agg2_minute")
+ sql(
+ s"""
+ | create datamap agg1 on table mainTable
--- End diff --
change datamap name to big_agg
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2118/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
Anyone can help to review it. @ravipesala @jackylk @QiangCai @chenliang613 @sraghunandan @kumarvishal09
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3001/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r165051297
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesMatchStrategySuite.scala ---
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesMatchStrategySuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+ override def beforeAll: Unit = {
+ SparkUtil4Test.createTaskMockUp(sqlContext)
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+
+ }
+
+ override protected def beforeEach(): Unit = {
+ sql("drop table if exists mainTable")
+ sql(
+ """
+ | CREATE TABLE mainTable(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
+ }
+
+ val timeSeries = "preaggregate"
+
+ test("test timeseries match 1: select small size when there are >1 timeseries match") {
+
+ dropDataMaps("maintable", "agg1_minute", "agg2_minute")
+ sql(
+ s"""
+ | create datamap agg1 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age), count(age), max(age), min(age)
+ | from mainTable
+ | group by mytime
+ """.stripMargin)
+
+ val df1 = sql(
+ """
+ | select
+ | timeseries(mytime,'minute') as minuteLevel,
+ | sum(age) as sum
+ | from mainTable
+ | where
+ | timeseries(mytime,'minute')<'2016-02-23 01:02:00' and
+ | timeseries(mytime,'minute')>='2016-02-23 01:01:00'
+ | group by
+ | timeseries(mytime,'minute')
+ | order by
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df1, Seq(Row(Timestamp.valueOf("2016-02-23 01:01:00"), 60)))
+
+ preAggTableValidator(df1.queryExecution.analyzed, "maintable_agg1_minute")
+
+ sql(
+ s"""
+ | create datamap agg2 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age)
+ | from mainTable
+ | group by mytime
+ """.stripMargin)
+
+ val df2 = sql(
+ """
+ | select
+ | timeseries(mytime,'minute') as minuteLevel,
+ | sum(age) as sum
+ | from mainTable
+ | where
+ | timeseries(mytime,'minute')<'2016-02-23 01:02:00' and
+ | timeseries(mytime,'minute')>='2016-02-23 01:01:00'
+ | group by
+ | timeseries(mytime,'minute')
+ | order by
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df2, Seq(Row(Timestamp.valueOf("2016-02-23 01:01:00"), 60)))
+
+ preAggTableValidator(df2.queryExecution.analyzed, "maintable_agg2_minute")
+ }
+
+ test("test timeseries match 2: select small size when there are >1 timeseries match") {
+ dropDataMaps("maintable", "agg1_minute", "agg2_minute")
+ sql(
+ s"""
+ | create datamap agg1 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age), count(age), max(age), min(age)
+ | from mainTable
+ | group by mytime
+ """.stripMargin)
+
+ sql(
+ s"""
+ | create datamap agg2 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age)
--- End diff --
ok, done
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174064693
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesDataLoad.scala ---
@@ -330,16 +366,232 @@ class TestTimeseriesDataLoad extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
checkAnswer(sql("select * from maintable_agg0_second"),
- Seq(Row(Timestamp.valueOf("2016-02-23 01:01:30.0"), 10),
- Row(Timestamp.valueOf("2016-02-23 01:01:40.0"), 20),
- Row(Timestamp.valueOf("2016-02-23 01:01:50.0"), 30),
- Row(Timestamp.valueOf("2016-02-23 01:02:30.0"), 40),
- Row(Timestamp.valueOf("2016-02-23 01:02:40.0"), 50),
- Row(Timestamp.valueOf("2016-02-23 01:02:50.0"), 50)))
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50)))
+ }
+
+ test("test timeseries table selection 14: load data into mainTable after create timeseries datamap ON TABLE and SELECT sub table") {
+ sql("DROP TABLE IF EXISTS main_table")
+ sql(
+ """
+ | CREATE TABLE main_table(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ sql(
+ s"""
+ | CREATE DATAMAP agg0_second ON TABLE main_table
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='mytime',
+ | 'SECOND_GRANULARITY'='1')
+ | AS SELECT mytime, SUM(age)
+ | FROM main_table
+ | GROUP BY mytime""".stripMargin)
+
+
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+
+ checkAnswer(sql("SELECT * FROM main_table_agg0_second"),
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50)))
+ }
+
+ test("test timeseries table selection 15: load data into main_table after create timeseries datamap ON TABLE 1") {
+ sql("DROP TABLE IF EXISTS main_table")
+ sql(
+ """
+ | CREATE TABLE main_table(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+
+ sql(
+ s"""
+ | CREATE DATAMAP agg0_minute ON TABLE main_table
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='mytime',
+ | 'MINUTE_GRANULARITY'='1')
+ | AS SELECT mytime, SUM(age)
+ | FROM main_table
+ | GROUP BY mytime""".stripMargin)
+
+ sql(s"LOAD DATA INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ val df = sql(
+ """
+ | SELECT
+ | timeseries(mytime,'minute') AS minuteLevel,
+ | SUM(age) AS SUM
+ | FROM main_table
+ | WHERE timeseries(mytime,'minute')>='2016-02-23 09:01:00'
+ | GROUP BY
+ | timeseries(mytime,'minute')
+ | ORDER BY
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df,
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 120),
+ Row(Timestamp.valueOf("2016-02-23 09:02:00"), 280)))
+ }
+
+ test("test timeseries table selection 16: load data into main_table after create timeseries datamap ON TABLE 2") {
+ sql("DROP TABLE IF EXISTS main_table")
+ sql(
+ """
+ | CREATE TABLE main_table(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+
+ sql(s"LOAD DATA INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ sql(s"LOAD DATA INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ sql(
+ s"""
+ | CREATE DATAMAP agg0_minute ON TABLE main_table
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='mytime',
+ | 'MINUTE_GRANULARITY'='1')
+ | AS SELECT mytime, SUM(age)
+ | FROM main_table
+ | GROUP BY mytime""".stripMargin)
+
+
+ val df = sql(
+ """
+ | SELECT
+ | timeseries(mytime,'minute') AS minuteLevel,
+ | SUM(age) AS SUM
+ | FROM main_table
+ | WHERE timeseries(mytime,'minute')>='2016-02-23 09:01:00'
+ | GROUP BY
+ | timeseries(mytime,'minute')
+ | ORDER BY
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df,
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 120),
+ Row(Timestamp.valueOf("2016-02-23 09:02:00"), 280)))
+ }
+
+ test("test timeseries table selection 17: load data into main_table after create timeseries datamap ON TABLE 3") {
--- End diff --
duplicate
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174404853
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -354,21 +390,137 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
checkExistence(sql("DESC FORMATTED mainTable_agg1"), true, "maintable_age_sum")
}
+ test("test timeseries create table 32: should support if not exists, create when same table not exists") {
+ sql("DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ sql(
+ s"""
+ |CREATE DATAMAP if not exists agg1_year ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'YEAR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_year")
+ checkExistence(sql("DESC FORMATTED mainTable_agg1_year"), true, "maintable_age_sum")
+ }
+
test("test timeseries create table 20: don't support 'create datamap if exists'") {
val e: Exception = intercept[AnalysisException] {
sql(
s"""CREATE DATAMAP IF EXISTS agg2 ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'MONTH_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'EVENT_TIME'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) FROM mainTable
+ | GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.contains("identifier matching regex"))
}
+ test("test timeseries create table 26: test different data type") {
+ sql("drop table if exists dataTable")
+ sql(
+ s"""
+ | CREATE TABLE dataTable(
+ | shortField SHORT,
+ | booleanField BOOLEAN,
+ | intField INT,
+ | bigintField LONG,
+ | doubleField DOUBLE,
+ | stringField STRING,
+ | decimalField DECIMAL(18,2),
+ | charField CHAR(5),
+ | floatField FLOAT,
+ | dataTime timestamp
+ | )
+ | STORED BY 'carbondata'
+ """.stripMargin)
+
+
+ sql(
+ s"""CREATE DATAMAP agg0_hour ON TABLE dataTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ | AS SELECT
+ | dataTime,
+ | SUM(intField),
+ | shortField,
+ | booleanField,
+ | intField,
+ | bigintField,
+ | doubleField,
+ | stringField,
+ | decimalField,
+ | charField,
+ | floatField
+ | FROM dataTable
+ | GROUP BY
+ | dataTime,
+ | shortField,
+ | booleanField,
+ | intField,
+ | bigintField,
+ | doubleField,
+ | stringField,
+ | decimalField,
+ | charField,
+ | floatField
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE dataTable"), true, "datatable_agg0_hour")
+ sql("DROP TABLE IF EXISTS dataTable")
+ }
+
+ test("test timeseries create table 27: test data map name") {
--- End diff --
but it has covered this scenario also right??
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3300/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2008/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3182/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4661/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3969/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4361/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
rebase again
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2392/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4189/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174064855
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesTableSelection.scala ---
@@ -99,124 +113,763 @@ class TestTimeseriesTableSelection extends QueryTest with BeforeAndAfterAll {
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
}
- test("test PreAggregate table selection 1") {
- val df = sql("select mytime from mainTable group by mytime")
+ test("test timeseries table selection 1") {
+ val df = sql("SELECT mytime FROM mainTable GROUP BY mytime")
preAggTableValidator(df.queryExecution.analyzed, "maintable")
}
- test("test PreAggregate table selection 2") {
- val df = sql("select timeseries(mytime,'hour') from mainTable group by timeseries(mytime,'hour')")
+ test("test timeseries table selection 2") {
+ val df = sql("SELECT TIMESERIES(mytime,'hour') FROM mainTable GROUP BY TIMESERIES(mytime,'hour')")
preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 3") {
- val df = sql("select timeseries(mytime,'milli') from mainTable group by timeseries(mytime,'milli')")
- preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ test("test timeseries table selection 3: No enum constant MILLI") {
+ val e = intercept[Exception] {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'milli')
+ | FROM mainTable
+ | GROUP BY TIMESERIES(mytime,'milli')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ df.show()
+ }
+ assert(e.getMessage.contains(
+ "No enum constant org.apache.carbondata.core.preagg.TimeSeriesFunctionEnum.MILLI"))
}
- test("test PreAggregate table selection 4") {
- val df = sql("select timeseries(mytime,'year') from mainTable group by timeseries(mytime,'year')")
+ test("test timeseries table selection 4") {
+ val df = sql("SELECT TIMESERIES(mytime,'year') FROM mainTable GROUP BY TIMESERIES(mytime,'year')")
--- End diff --
why is this requried?
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@jackylk Please review it 。 @kumarvishal09 @sraghunandan
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2435/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2273/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1871/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4589/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@jackylk Please review again
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest sdv please
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3075/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2307/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4245/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174406656
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -17,18 +17,28 @@
package org.apache.carbondata.integration.spark.testsuite.timeseries
import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
import org.apache.spark.sql.test.util.QueryTest
-import org.scalatest.BeforeAndAfterAll
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.common.exceptions.sql.{MalformedCarbonCommandException, MalformedDataMapCommandException}
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.datamap.DataMapProvider.TIMESERIES
+import org.apache.carbondata.core.util.CarbonProperties
-class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
+class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach{
val timeSeries = TIMESERIES.toString
+ var timestampFormat: String = _
--- End diff --
ok
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3088/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4385/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3508/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2375/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1861/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3537/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2301/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174130738
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala ---
@@ -33,16 +33,15 @@ class TestSortColumns extends QueryTest with BeforeAndAfterAll {
CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "dd-MM-yyyy")
SparkUtil4Test.createTaskMockUp(sqlContext)
- dropTable
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "dd-MM-yyyy")
+ dropTestTables
--- End diff --
I add common method QueryTest for all test case in org.apache.spark.sql.test.util.QueryTest, so subclass should change the method name
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3409/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
Please review it @kunal642
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r164659399
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesMatchStrategySuite.scala ---
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesMatchStrategySuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+ override def beforeAll: Unit = {
+ SparkUtil4Test.createTaskMockUp(sqlContext)
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+
+ }
+
+ override protected def beforeEach(): Unit = {
+ sql("drop table if exists mainTable")
+ sql(
+ """
+ | CREATE TABLE mainTable(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
+ }
+
+ val timeSeries = "preaggregate"
+
+ test("test timeseries match 1: select small size when there are >1 timeseries match") {
+
+ dropDataMaps("maintable", "agg1_minute", "agg2_minute")
+ sql(
+ s"""
+ | create datamap agg1 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age), count(age), max(age), min(age)
+ | from mainTable
+ | group by mytime
+ """.stripMargin)
+
+ val df1 = sql(
+ """
+ | select
+ | timeseries(mytime,'minute') as minuteLevel,
+ | sum(age) as sum
+ | from mainTable
+ | where
+ | timeseries(mytime,'minute')<'2016-02-23 01:02:00' and
+ | timeseries(mytime,'minute')>='2016-02-23 01:01:00'
+ | group by
+ | timeseries(mytime,'minute')
+ | order by
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df1, Seq(Row(Timestamp.valueOf("2016-02-23 01:01:00"), 60)))
+
+ preAggTableValidator(df1.queryExecution.analyzed, "maintable_agg1_minute")
+
+ sql(
+ s"""
+ | create datamap agg2 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age)
+ | from mainTable
+ | group by mytime
+ """.stripMargin)
+
+ val df2 = sql(
+ """
+ | select
+ | timeseries(mytime,'minute') as minuteLevel,
+ | sum(age) as sum
+ | from mainTable
+ | where
+ | timeseries(mytime,'minute')<'2016-02-23 01:02:00' and
+ | timeseries(mytime,'minute')>='2016-02-23 01:01:00'
+ | group by
+ | timeseries(mytime,'minute')
+ | order by
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df2, Seq(Row(Timestamp.valueOf("2016-02-23 01:01:00"), 60)))
+
+ preAggTableValidator(df2.queryExecution.analyzed, "maintable_agg2_minute")
+ }
+
+ test("test timeseries match 2: select small size when there are >1 timeseries match") {
+ dropDataMaps("maintable", "agg1_minute", "agg2_minute")
+ sql(
+ s"""
+ | create datamap agg1 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age), count(age), max(age), min(age)
+ | from mainTable
+ | group by mytime
+ """.stripMargin)
+
+ sql(
+ s"""
+ | create datamap agg2 on table mainTable
+ | using '$timeSeries'
+ | DMPROPERTIES (
+ | 'timeseries.eventTime'='mytime',
+ | 'timeseries.hierarchy'='minute=1')
+ | as select mytime, sum(age)
--- End diff --
should create this datamap first, and modify testcase title
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3434/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174119683
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -82,149 +92,209 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ override def afterEach(): Unit = {
+ dropDataMaps("mainTable", "agg1_second", "agg1_minute",
+ "agg1_hour", "agg1_day", "agg1_month", "agg1_year")
+ }
+
test("test timeseries create table 1") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_second"), true, "maintable_agg0_second")
- sql("drop datamap agg0_second on table mainTable")
+ sql("DROP DATAMAP agg0_second ON TABLE mainTable")
}
test("test timeseries create table 2") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_hour"), true, "maintable_agg0_hour")
- sql("drop datamap agg0_hour on table mainTable")
+ sql("DROP DATAMAP agg0_hour ON TABLE mainTable")
}
+
test("test timeseries create table 3") {
checkExistence(sql("DESCRIBE FORMATTED maintable_agg0_day"), true, "maintable_agg0_day")
- sql("drop datamap agg0_day on table mainTable")
+ sql("DROP DATAMAP agg0_day ON TABLE mainTable")
}
+
test("test timeseries create table 4") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_month"), true, "maintable_agg0_month")
- sql("drop datamap agg0_month on table mainTable")
+ sql("DROP DATAMAP agg0_month ON TABLE mainTable")
}
+
test("test timeseries create table 5") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_year"), true, "maintable_agg0_year")
- sql("drop datamap agg0_year on table mainTable")
+ sql("DROP DATAMAP agg0_year ON TABLE mainTable")
}
- test("test timeseries create table 6") {
- intercept[Exception] {
+ test("test timeseries create table 6: TIMESERIES should define time granularity") {
+ sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
- s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'SEC_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ s"""CREATE DATAMAP agg0_second ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'SEC_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
""".stripMargin)
}
+ assert(e.getMessage.contains("TIMESERIES should define time granularity"))
}
- test("test timeseries create table 7") {
- intercept[Exception] {
+ test("test timeseries create table 7: Granularity only support 1") {
+ sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ val e = intercept[MalformedDataMapCommandException] {
sql(
s"""
| CREATE DATAMAP agg0_second ON TABLE mainTable
| USING '$timeSeries'
| DMPROPERTIES (
| 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='2')
+ | 'DAY_GRANULARITY'='1.5')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
+ assert(e.getMessage.equals("Granularity only support 1"))
}
- test("test timeseries create table 8") {
- intercept[Exception] {
+ test("test timeseries create table 8: Granularity only support 1") {
+ dropDataMaps("mainTable", "agg1_hour")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
- s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
+ s"""CREATE DATAMAP agg1_hour ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='hour=-2')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ }
+ assert(e.getMessage.contains("Granularity only support "))
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "maintable_agg1_hour")
+ }
+
+ test("test timeseries create table 9: SECOND_GRANULARITY is null") {
+ sql("DROP DATAMAP IF EXISTS agg1 ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
- s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
+ s"""CREATE DATAMAP agg0_hour ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ }
+ assert(e.getMessage.contains("Granularity only support 1"))
+ }
+
+ test("test timeseries create table 10: Table already exists in database") {
+ val e = intercept[MalformedDataMapCommandException] {
+ sql(
+ s"""CREATE DATAMAP agg1_hour ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ sql(
+ s"""CREATE DATAMAP agg1_hour ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
}
+ assert(e.getMessage.contains(
+ "DataMap name 'agg1_hour' already exist"))
}
- test("test timeseries create table 9") {
- intercept[Exception] {
+ test("test timeseries create table 11: don't support create timeseries table on non timestamp") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
+ | CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING '$timeSeries'
| DMPROPERTIES (
| 'EVENT_TIME'='name',
- | 'SECOND_GRANULARITY'='1')
+ | 'HOUR_GRANULARITY'='1')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
+ assert(e.getMessage.equals("Timeseries event time is only supported on Timestamp column"))
}
- test("test timeseries create table 10") {
- intercept[Exception] {
+ test("test timeseries create table 12: Time series column dataTime does not exists in select") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
+ | CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING '$timeSeries'
| DMPROPERTIES (
- | 'EVENT_TIME'='name',
- | 'SECOND_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
+ | 'EVENT_TIME'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ | AS SELECT name, SUM(age) FROM mainTable
+ | GROUP BY name
+ """.stripMargin)
+ }
+ assert(e.getMessage.equals("Time series column dataTime does not exists in select"))
+ }
+
+ test("test timeseries create table 13: don't support create timeseries table on non timestamp") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
+ sql(
+ s"""CREATE DATAMAP agg0_hour ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='name',
+ | 'HOUR_GRANULARITY'='1')
+ |AS SELECT name, SUM(age) FROM mainTable
+ |GROUP BY name
+ """.stripMargin)
}
+ assert(e.getMessage.contains("Timeseries event time is only supported on Timestamp column"))
}
- test("test timeseries create table 11: USING") {
+ test("test timeseries create table 14: USING") {
val e: Exception = intercept[MalformedDataMapCommandException] {
sql(
- """CREATE DATAMAP agg1 ON TABLE mainTable
+ """CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING 'abc'
| DMPROPERTIES (
| 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
+ | 'HOUR_GRANULARITY'='1')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.equals("DataMap 'abc' not found"))
}
- test("test timeseries create table 12: USING and catch MalformedCarbonCommandException") {
+ test("test timeseries create table 15: USING and catch MalformedCarbonCommandException") {
val e: Exception = intercept[MalformedCarbonCommandException] {
sql(
- """CREATE DATAMAP agg1 ON TABLE mainTable
+ """CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING 'abc'
| DMPROPERTIES (
| 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
+ | 'HOUR_GRANULARITY'='1')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.equals("DataMap 'abc' not found"))
}
- test("test timeseries create table 13: Only one granularity level can be defined 1") {
- sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ test("test timeseries create table 16: Only one granularity level can be defined 1") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
val e: Exception = intercept[MalformedCarbonCommandException] {
sql(
s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
+ | CREATE DATAMAP agg0_hour ON TABLE mainTable
--- End diff --
It's unreasonable to create pre-aggregate with second granularity, second aggregate is used less than hour in reality life.
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by sraghunandan <gi...@git.apache.org>.
Github user sraghunandan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r167120975
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesDropSuite.scala ---
@@ -36,16 +48,168 @@ class TestTimeSeriesDropSuite extends QueryTest with BeforeAndAfterAll with Befo
""".stripMargin)
}
- test("test timeseries drop datamap 1: drop datamap should throw exception if no datamap") {
+ override def afterEach(): Unit = {
+ dropDataMaps("mainTable", "agg1_second", "agg1_minute",
+ "agg1_hour", "agg1_day", "agg1_month", "agg1_year")
+ }
+
+ test("test timeseries drop datamap 1: drop datamap should throw exception") {
// DROP DATAMAP DataMapName if the DataMapName not exists
checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
val e: Exception = intercept[Exception] {
sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
}
- assert(e.getMessage.equals("Datamap with name agg1_month does not exist under table mainTable"))
+ assert(e.getMessage.contains(
+ "Datamap with name agg1_month does not exist under table mainTable"))
+ }
+
+ test("test timeseries drop datamap 2: should support drop datamap IF EXISTS") {
+ // DROP DATAMAP IF EXISTS DataMapName
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
+ sql(s"DROP DATAMAP IF EXISTS agg1_month ON TABLE mainTable")
+ assert(true)
+ }
+
+ test("test timeseries drop datamap 3: should support drop datamap") {
+ sql(
+ s"""
+ | CREATE DATAMAP agg1_month ON TABLE mainTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) from mainTable
+ | GROUP BY dataTime
+ """.stripMargin)
+
+ // Before DROP DATAMAP
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_month")
+
+ // DROP DATAMAP DataMapName
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
+ val e: Exception = intercept[Exception] {
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ }
+ assert(e.getMessage.contains(
+ "Datamap with name agg1_month does not exist under table mainTable"))
+ }
+
+ test("test timeseries drop datamap 4: should support drop datamap with IF EXISTS") {
+ sql(
+ s"""
+ | CREATE DATAMAP agg1_month ON TABLE mainTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) from mainTable
+ | GROUP BY dataTime
+ """.stripMargin)
+ // DROP DATAMAP IF EXISTS DataMapName
+ sql(s"DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ checkExistence(sql("SHOW DATAMAP ON TABLE maintable"), false, "agg1_year")
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_month")
+
+ // DROP DATAMAP DataMapName
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
+ val e: Exception = intercept[Exception] {
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ }
+ assert(e.getMessage.contains(
+ "Datamap with name agg1_month does not exist under table mainTable"))
+ }
+
+ test("test timeseries drop datamap 5: drop datamap when table not exists") {
--- End diff --
Modify the existing test case to catch MalformedCarbonCommandException
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174126731
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -354,21 +390,137 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
checkExistence(sql("DESC FORMATTED mainTable_agg1"), true, "maintable_age_sum")
}
+ test("test timeseries create table 32: should support if not exists, create when same table not exists") {
+ sql("DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ sql(
+ s"""
+ |CREATE DATAMAP if not exists agg1_year ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'YEAR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_year")
+ checkExistence(sql("DESC FORMATTED mainTable_agg1_year"), true, "maintable_age_sum")
+ }
+
test("test timeseries create table 20: don't support 'create datamap if exists'") {
val e: Exception = intercept[AnalysisException] {
sql(
s"""CREATE DATAMAP IF EXISTS agg2 ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'MONTH_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'EVENT_TIME'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) FROM mainTable
+ | GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.contains("identifier matching regex"))
}
+ test("test timeseries create table 26: test different data type") {
+ sql("drop table if exists dataTable")
+ sql(
+ s"""
+ | CREATE TABLE dataTable(
+ | shortField SHORT,
+ | booleanField BOOLEAN,
+ | intField INT,
+ | bigintField LONG,
+ | doubleField DOUBLE,
+ | stringField STRING,
+ | decimalField DECIMAL(18,2),
+ | charField CHAR(5),
+ | floatField FLOAT,
+ | dataTime timestamp
+ | )
+ | STORED BY 'carbondata'
+ """.stripMargin)
+
+
+ sql(
+ s"""CREATE DATAMAP agg0_hour ON TABLE dataTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ | AS SELECT
+ | dataTime,
+ | SUM(intField),
+ | shortField,
+ | booleanField,
+ | intField,
+ | bigintField,
+ | doubleField,
+ | stringField,
+ | decimalField,
+ | charField,
+ | floatField
+ | FROM dataTable
+ | GROUP BY
+ | dataTime,
+ | shortField,
+ | booleanField,
+ | intField,
+ | bigintField,
+ | doubleField,
+ | stringField,
+ | decimalField,
+ | charField,
+ | floatField
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE dataTable"), true, "datatable_agg0_hour")
+ sql("DROP TABLE IF EXISTS dataTable")
+ }
+
+ test("test timeseries create table 27: test data map name") {
--- End diff --
test case 32 and test 27 is different, test case 32 is check carbon should support if CREATE DATAMAP not exists
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174122755
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -82,149 +92,209 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ override def afterEach(): Unit = {
+ dropDataMaps("mainTable", "agg1_second", "agg1_minute",
+ "agg1_hour", "agg1_day", "agg1_month", "agg1_year")
+ }
+
test("test timeseries create table 1") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_second"), true, "maintable_agg0_second")
- sql("drop datamap agg0_second on table mainTable")
+ sql("DROP DATAMAP agg0_second ON TABLE mainTable")
--- End diff --
The operation should be upper case and table name should be down case according to SQL syntax, Hive syntax also like this.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1856
LGTM
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174060602
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesDropSuite.scala ---
@@ -16,15 +16,28 @@
*/
package org.apache.carbondata.integration.spark.testsuite.timeseries
+import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.test.util.QueryTest
import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
-import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
+import org.apache.carbondata.common.exceptions.sql.{MalformedCarbonCommandException, NoSuchDataMapException}
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.spark.exception.ProcessMetaDataException
class TestTimeSeriesDropSuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+ val timeSeries = "timeseries"
+ var timestampFormat: String = _
--- End diff --
no need for this variable. simply set CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT as the timestamp format is after all.
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174123530
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -17,18 +17,28 @@
package org.apache.carbondata.integration.spark.testsuite.timeseries
import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
import org.apache.spark.sql.test.util.QueryTest
-import org.scalatest.BeforeAndAfterAll
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.common.exceptions.sql.{MalformedCarbonCommandException, MalformedDataMapCommandException}
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.datamap.DataMapProvider.TIMESERIES
+import org.apache.carbondata.core.util.CarbonProperties
-class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
+class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach{
val timeSeries = TIMESERIES.toString
+ var timestampFormat: String = _
--- End diff --
@sraghunandan suggested should not change the CarbonCommonConstants values after running the test case, because other place maybe need this constants value.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3498/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3887/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r178216676
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesDataLoad.scala ---
@@ -167,59 +170,60 @@ class TestTimeseriesDataLoad extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
- test("test Year level timeseries data validation1 ") {
- checkAnswer( sql("select count(*) from table_03_ag1_year"),
+
+ test("test timeseries table selection 1: year level timeseries data validation1 ") {
+ checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_year"),
Seq(Row(4)))
}
- test("test month level timeseries data validation1 ") {
- checkAnswer( sql("select count(*) from table_03_ag1_month"),
+ test("test timeseries table selection 2: month level timeseries data validation1 ") {
+ checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_month"),
Seq(Row(4)))
}
- test("test day level timeseries data validation1 ") {
- checkAnswer( sql("select count(*) from table_03_ag1_day"),
+ test("test timeseries table selection 3: day level timeseries data validation1 ") {
+ checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_day"),
--- End diff --
I confirmed with jacky and raghu
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174128505
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesUnsupportedSuite.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesUnsupportedSuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
--- End diff --
Can you tell me where is the test case of inser/load/Update/delete/alter into timeseries aggregate table?
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3473/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174064625
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesDataLoad.scala ---
@@ -330,16 +366,232 @@ class TestTimeseriesDataLoad extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
checkAnswer(sql("select * from maintable_agg0_second"),
- Seq(Row(Timestamp.valueOf("2016-02-23 01:01:30.0"), 10),
- Row(Timestamp.valueOf("2016-02-23 01:01:40.0"), 20),
- Row(Timestamp.valueOf("2016-02-23 01:01:50.0"), 30),
- Row(Timestamp.valueOf("2016-02-23 01:02:30.0"), 40),
- Row(Timestamp.valueOf("2016-02-23 01:02:40.0"), 50),
- Row(Timestamp.valueOf("2016-02-23 01:02:50.0"), 50)))
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50)))
+ }
+
+ test("test timeseries table selection 14: load data into mainTable after create timeseries datamap ON TABLE and SELECT sub table") {
+ sql("DROP TABLE IF EXISTS main_table")
+ sql(
+ """
+ | CREATE TABLE main_table(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ sql(
+ s"""
+ | CREATE DATAMAP agg0_second ON TABLE main_table
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='mytime',
+ | 'SECOND_GRANULARITY'='1')
+ | AS SELECT mytime, SUM(age)
+ | FROM main_table
+ | GROUP BY mytime""".stripMargin)
+
+
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+
+ checkAnswer(sql("SELECT * FROM main_table_agg0_second"),
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50)))
+ }
+
+ test("test timeseries table selection 15: load data into main_table after create timeseries datamap ON TABLE 1") {
+ sql("DROP TABLE IF EXISTS main_table")
+ sql(
+ """
+ | CREATE TABLE main_table(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+
+ sql(
+ s"""
+ | CREATE DATAMAP agg0_minute ON TABLE main_table
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='mytime',
+ | 'MINUTE_GRANULARITY'='1')
+ | AS SELECT mytime, SUM(age)
+ | FROM main_table
+ | GROUP BY mytime""".stripMargin)
+
+ sql(s"LOAD DATA INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ val df = sql(
+ """
+ | SELECT
+ | timeseries(mytime,'minute') AS minuteLevel,
+ | SUM(age) AS SUM
+ | FROM main_table
+ | WHERE timeseries(mytime,'minute')>='2016-02-23 09:01:00'
+ | GROUP BY
+ | timeseries(mytime,'minute')
+ | ORDER BY
+ | timeseries(mytime,'minute')
+ """.stripMargin)
+
+ checkAnswer(df,
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 120),
+ Row(Timestamp.valueOf("2016-02-23 09:02:00"), 280)))
+ }
+
+ test("test timeseries table selection 16: load data into main_table after create timeseries datamap ON TABLE 2") {
--- End diff --
duplicate
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2232/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
need rebase again
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3459/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3632/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174060559
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -17,18 +17,28 @@
package org.apache.carbondata.integration.spark.testsuite.timeseries
import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
import org.apache.spark.sql.test.util.QueryTest
-import org.scalatest.BeforeAndAfterAll
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.common.exceptions.sql.{MalformedCarbonCommandException, MalformedDataMapCommandException}
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.metadata.schema.datamap.DataMapProvider.TIMESERIES
+import org.apache.carbondata.core.util.CarbonProperties
-class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
+class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach{
val timeSeries = TIMESERIES.toString
+ var timestampFormat: String = _
--- End diff --
no need for this variable. simply set CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT as the timestamp format is after all.
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174056762
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -82,149 +92,209 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ override def afterEach(): Unit = {
+ dropDataMaps("mainTable", "agg1_second", "agg1_minute",
+ "agg1_hour", "agg1_day", "agg1_month", "agg1_year")
+ }
+
test("test timeseries create table 1") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_second"), true, "maintable_agg0_second")
- sql("drop datamap agg0_second on table mainTable")
+ sql("DROP DATAMAP agg0_second ON TABLE mainTable")
--- End diff --
why is this change necessary??
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1967/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3295/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3242/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1904/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
we need rebase because https://github.com/apache/carbondata/commit/ee1c4d42fc0837e515ac222c676bd46fe93795d5
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1856
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3288/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1969/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174064354
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesDataLoad.scala ---
@@ -167,59 +170,60 @@ class TestTimeseriesDataLoad extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
- test("test Year level timeseries data validation1 ") {
- checkAnswer( sql("select count(*) from table_03_ag1_year"),
+
+ test("test timeseries table selection 1: year level timeseries data validation1 ") {
+ checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_year"),
Seq(Row(4)))
}
- test("test month level timeseries data validation1 ") {
- checkAnswer( sql("select count(*) from table_03_ag1_month"),
+ test("test timeseries table selection 2: month level timeseries data validation1 ") {
+ checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_month"),
Seq(Row(4)))
}
- test("test day level timeseries data validation1 ") {
- checkAnswer( sql("select count(*) from table_03_ag1_day"),
+ test("test timeseries table selection 3: day level timeseries data validation1 ") {
+ checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_day"),
--- End diff --
why is this required?
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174128639
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesTableSelection.scala ---
@@ -99,124 +113,763 @@ class TestTimeseriesTableSelection extends QueryTest with BeforeAndAfterAll {
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
}
- test("test PreAggregate table selection 1") {
- val df = sql("select mytime from mainTable group by mytime")
+ test("test timeseries table selection 1") {
+ val df = sql("SELECT mytime FROM mainTable GROUP BY mytime")
preAggTableValidator(df.queryExecution.analyzed, "maintable")
}
- test("test PreAggregate table selection 2") {
- val df = sql("select timeseries(mytime,'hour') from mainTable group by timeseries(mytime,'hour')")
+ test("test timeseries table selection 2") {
+ val df = sql("SELECT TIMESERIES(mytime,'hour') FROM mainTable GROUP BY TIMESERIES(mytime,'hour')")
preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 3") {
- val df = sql("select timeseries(mytime,'milli') from mainTable group by timeseries(mytime,'milli')")
- preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ test("test timeseries table selection 3: No enum constant MILLI") {
+ val e = intercept[Exception] {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'milli')
+ | FROM mainTable
+ | GROUP BY TIMESERIES(mytime,'milli')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ df.show()
+ }
+ assert(e.getMessage.contains(
+ "No enum constant org.apache.carbondata.core.preagg.TimeSeriesFunctionEnum.MILLI"))
}
- test("test PreAggregate table selection 4") {
- val df = sql("select timeseries(mytime,'year') from mainTable group by timeseries(mytime,'year')")
+ test("test timeseries table selection 4") {
+ val df = sql("SELECT TIMESERIES(mytime,'year') FROM mainTable GROUP BY TIMESERIES(mytime,'year')")
--- End diff --
The operation should be upper case and table name should be down case according to SQL syntax, Hive syntax also like this.
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3073/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2399/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3534/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3856/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4171/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@jackylk Please review it
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@jackylk Please review
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2298/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r167147440
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesDropSuite.scala ---
@@ -36,16 +48,168 @@ class TestTimeSeriesDropSuite extends QueryTest with BeforeAndAfterAll with Befo
""".stripMargin)
}
- test("test timeseries drop datamap 1: drop datamap should throw exception if no datamap") {
+ override def afterEach(): Unit = {
+ dropDataMaps("mainTable", "agg1_second", "agg1_minute",
+ "agg1_hour", "agg1_day", "agg1_month", "agg1_year")
+ }
+
+ test("test timeseries drop datamap 1: drop datamap should throw exception") {
// DROP DATAMAP DataMapName if the DataMapName not exists
checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
val e: Exception = intercept[Exception] {
sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
}
- assert(e.getMessage.equals("Datamap with name agg1_month does not exist under table mainTable"))
+ assert(e.getMessage.contains(
+ "Datamap with name agg1_month does not exist under table mainTable"))
+ }
+
+ test("test timeseries drop datamap 2: should support drop datamap IF EXISTS") {
+ // DROP DATAMAP IF EXISTS DataMapName
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
+ sql(s"DROP DATAMAP IF EXISTS agg1_month ON TABLE mainTable")
+ assert(true)
+ }
+
+ test("test timeseries drop datamap 3: should support drop datamap") {
+ sql(
+ s"""
+ | CREATE DATAMAP agg1_month ON TABLE mainTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) from mainTable
+ | GROUP BY dataTime
+ """.stripMargin)
+
+ // Before DROP DATAMAP
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_month")
+
+ // DROP DATAMAP DataMapName
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
+ val e: Exception = intercept[Exception] {
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ }
+ assert(e.getMessage.contains(
+ "Datamap with name agg1_month does not exist under table mainTable"))
+ }
+
+ test("test timeseries drop datamap 4: should support drop datamap with IF EXISTS") {
+ sql(
+ s"""
+ | CREATE DATAMAP agg1_month ON TABLE mainTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) from mainTable
+ | GROUP BY dataTime
+ """.stripMargin)
+ // DROP DATAMAP IF EXISTS DataMapName
+ sql(s"DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ checkExistence(sql("SHOW DATAMAP ON TABLE maintable"), false, "agg1_year")
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_month")
+
+ // DROP DATAMAP DataMapName
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "agg1_month")
+ val e: Exception = intercept[Exception] {
+ sql(s"DROP DATAMAP agg1_month ON TABLE mainTable")
+ }
+ assert(e.getMessage.contains(
+ "Datamap with name agg1_month does not exist under table mainTable"))
+ }
+
+ test("test timeseries drop datamap 5: drop datamap when table not exists") {
--- End diff --
ok,done
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3197/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174127244
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesMatchStrategySuite.scala ---
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.integration.spark.testsuite.timeseries
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestTimeSeriesMatchStrategySuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+
+ var timestampFormat: String = _
--- End diff --
@sraghunandan suggested should not change the CarbonCommonConstants values after running the test case, because other place maybe need this constants value.
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174119980
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -241,12 +311,12 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
assert(e.getMessage.equals("Only one granularity level can be defined"))
}
- test("test timeseries create table 14: Only one granularity level can be defined 2") {
- sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ test("test timeseries create table 17: Only one granularity level can be defined 2") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
--- End diff --
It's unreasonable to create pre-aggregate with second granularity, second aggregate is used less than hour in reality life.
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3089/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2149/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3127/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174059512
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -354,21 +390,137 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
checkExistence(sql("DESC FORMATTED mainTable_agg1"), true, "maintable_age_sum")
}
+ test("test timeseries create table 32: should support if not exists, create when same table not exists") {
+ sql("DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ sql(
+ s"""
+ |CREATE DATAMAP if not exists agg1_year ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'YEAR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_year")
+ checkExistence(sql("DESC FORMATTED mainTable_agg1_year"), true, "maintable_age_sum")
+ }
+
test("test timeseries create table 20: don't support 'create datamap if exists'") {
val e: Exception = intercept[AnalysisException] {
sql(
s"""CREATE DATAMAP IF EXISTS agg2 ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'MONTH_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'EVENT_TIME'='dataTime',
+ | 'MONTH_GRANULARITY'='1')
+ | AS SELECT dataTime, SUM(age) FROM mainTable
+ | GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.contains("identifier matching regex"))
}
+ test("test timeseries create table 26: test different data type") {
+ sql("drop table if exists dataTable")
+ sql(
+ s"""
+ | CREATE TABLE dataTable(
+ | shortField SHORT,
+ | booleanField BOOLEAN,
+ | intField INT,
+ | bigintField LONG,
+ | doubleField DOUBLE,
+ | stringField STRING,
+ | decimalField DECIMAL(18,2),
+ | charField CHAR(5),
+ | floatField FLOAT,
+ | dataTime timestamp
+ | )
+ | STORED BY 'carbondata'
+ """.stripMargin)
+
+
+ sql(
+ s"""CREATE DATAMAP agg0_hour ON TABLE dataTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ | AS SELECT
+ | dataTime,
+ | SUM(intField),
+ | shortField,
+ | booleanField,
+ | intField,
+ | bigintField,
+ | doubleField,
+ | stringField,
+ | decimalField,
+ | charField,
+ | floatField
+ | FROM dataTable
+ | GROUP BY
+ | dataTime,
+ | shortField,
+ | booleanField,
+ | intField,
+ | bigintField,
+ | doubleField,
+ | stringField,
+ | decimalField,
+ | charField,
+ | floatField
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE dataTable"), true, "datatable_agg0_hour")
+ sql("DROP TABLE IF EXISTS dataTable")
+ }
+
+ test("test timeseries create table 27: test data map name") {
--- End diff --
same as test timeseries create table 32. Please remove. avoid adding duplicate test cases
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
@kunal642 Done, please check again. Please tell me if there are any problem. Can you help me to merge it if it's no problem? I spend much time to rebase before.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3472/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4191/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174066445
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesTableSelection.scala ---
@@ -99,124 +113,763 @@ class TestTimeseriesTableSelection extends QueryTest with BeforeAndAfterAll {
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
}
- test("test PreAggregate table selection 1") {
- val df = sql("select mytime from mainTable group by mytime")
+ test("test timeseries table selection 1") {
+ val df = sql("SELECT mytime FROM mainTable GROUP BY mytime")
preAggTableValidator(df.queryExecution.analyzed, "maintable")
}
- test("test PreAggregate table selection 2") {
- val df = sql("select timeseries(mytime,'hour') from mainTable group by timeseries(mytime,'hour')")
+ test("test timeseries table selection 2") {
+ val df = sql("SELECT TIMESERIES(mytime,'hour') FROM mainTable GROUP BY TIMESERIES(mytime,'hour')")
preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 3") {
- val df = sql("select timeseries(mytime,'milli') from mainTable group by timeseries(mytime,'milli')")
- preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ test("test timeseries table selection 3: No enum constant MILLI") {
+ val e = intercept[Exception] {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'milli')
+ | FROM mainTable
+ | GROUP BY TIMESERIES(mytime,'milli')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ df.show()
+ }
+ assert(e.getMessage.contains(
+ "No enum constant org.apache.carbondata.core.preagg.TimeSeriesFunctionEnum.MILLI"))
}
- test("test PreAggregate table selection 4") {
- val df = sql("select timeseries(mytime,'year') from mainTable group by timeseries(mytime,'year')")
+ test("test timeseries table selection 4") {
+ val df = sql("SELECT TIMESERIES(mytime,'year') FROM mainTable GROUP BY TIMESERIES(mytime,'year')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_year")
}
- test("test PreAggregate table selection 5") {
- val df = sql("select timeseries(mytime,'day') from mainTable group by timeseries(mytime,'day')")
+ test("test timeseries table selection 5") {
+ val df = sql("SELECT TIMESERIES(mytime,'day') FROM mainTable GROUP BY TIMESERIES(mytime,'day')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_day")
}
- test("test PreAggregate table selection 6") {
- val df = sql("select timeseries(mytime,'month') from mainTable group by timeseries(mytime,'month')")
+ test("test timeseries table selection 6") {
+ val df = sql("SELECT TIMESERIES(mytime,'month') FROM mainTable GROUP BY TIMESERIES(mytime,'month')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_month")
}
- test("test PreAggregate table selection 7") {
- val df = sql("select timeseries(mytime,'minute') from mainTable group by timeseries(mytime,'minute')")
+ test("test timeseries table selection 7") {
+ val df = sql("SELECT TIMESERIES(mytime,'minute') FROM mainTable GROUP BY TIMESERIES(mytime,'minute')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_minute")
}
- test("test PreAggregate table selection 8") {
- val df = sql("select timeseries(mytime,'second') from mainTable group by timeseries(mytime,'second')")
+ test("test timeseries table selection 8") {
+ val df = sql("SELECT TIMESERIES(mytime,'second') FROM mainTable GROUP BY TIMESERIES(mytime,'second')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_second")
}
- test("test PreAggregate table selection 9") {
- val df = sql("select timeseries(mytime,'hour') from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 9") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 10") {
- val df = sql("select timeseries(mytime,'hour') from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 10") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
+ }
+
+ test("test timeseries table selection 11") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour'),SUM(age)
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 11") {
- val df = sql("select timeseries(mytime,'hour'),sum(age) from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 12") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour') AS hourlevel,SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 12") {
- val df = sql("select timeseries(mytime,'hour')as hourlevel,sum(age) as sum from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 13") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')as hourlevel,SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x' AND name='vishal'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
}
- test("test PreAggregate table selection 13") {
- val df = sql("select timeseries(mytime,'hour')as hourlevel,sum(age) as sum from mainTable where timeseries(mytime,'hour')='x' and name='vishal' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable")
+ test("test timeseries table selection 14: TIMESERIES(mytime,'hour') match") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='2016-02-23 09:00:00'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
+ checkAnswer(df, Row(Timestamp.valueOf("2016-02-23 09:00:00.0")))
}
- test("test timeseries table selection 14: Granularity only support 1 and throw Exception") {
- val e = intercept[MalformedCarbonCommandException] {
- sql(
- s"""
- | CREATE DATAMAP agg3_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'HOUR_GRANULARITY'='2')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
- }
- assert(e.getMessage.contains("Granularity only support 1"))
- }
-
- test("test timeseries table selection 15: Granularity only support 1 and throw Exception") {
- val e = intercept[MalformedCarbonCommandException] {
- sql(
- s"""
- | CREATE DATAMAP agg3_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'HOUR_GRANULARITY'='1.5')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
- }
- assert(e.getMessage.contains("Granularity only support 1"))
- }
-
- test("test timeseries table selection 16: Granularity only support 1 and throw Exception") {
- val e = intercept[MalformedCarbonCommandException] {
- sql(
- s"""
- | CREATE DATAMAP agg3_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'HOUR_GRANULARITY'='-1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
- }
- assert(e.getMessage.contains("Granularity only support 1"))
+ test("test timeseries table selection 15: TIMESERIES(mytime,'hour') not match") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='2016-02-23 09:01:00'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
+ checkExistence(df, false, "2016-02-23 09:00:00", "2016-02-23 09:01:00")
+ }
+
+ test("test timeseries table selection 16: TIMESERIES(mytime,'minute') match") {
+ checkExistence(sql("SELECT * FROM mainTable"), true,
+ "2016-02-23 09:01:30", "2016-02-23 09:02:40")
+ checkExistence(sql("SELECT * FROM mainTable"), false,
+ "2016-02-23 09:02:00", "2016-02-23 09:01:00")
+ val df = sql(
+ """
+ |SELECT TIMESERIES(mytime,'minute')
+ |FROM mainTable
+ |GROUP BY TIMESERIES(mytime,'minute')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_minute")
+ checkExistence(df, true, "2016-02-23 09:02:00", "2016-02-23 09:01:00")
+ checkAnswer(df,
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:02:00.0")),
+ Row(Timestamp.valueOf("2016-02-23 09:01:00.0"))))
+
+ val df2 = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'minute')as minutelevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'minute')='2016-02-23 09:01:00'
+ | GROUP BY TIMESERIES(mytime,'minute')
+ | ORDER BY TIMESERIES(mytime,'minute')
+ """.stripMargin)
+ preAggTableValidator(df2.queryExecution.analyzed, "maintable_agg0_minute")
+ checkAnswer(df2, Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 60)))
+ }
+
+ test("test timeseries table selection 17: TIMESERIES(mytime,'minute') not match pre agg") {
+ val df = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'minute')as minutelevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'minute')='2016-02-23 09:01:00' AND name='vishal'
+ | GROUP BY TIMESERIES(mytime,'minute')
+ | ORDER BY TIMESERIES(mytime,'minute')
+ """.stripMargin)
+ checkAnswer(df, Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 10)))
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ }
+
+ test("test timeseries table selection 18: select with many GROUP BY AND one filter") {
+ val df = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'year') AS yearLevel,
+ | TIMESERIES(mytime,'month') AS monthLevel,
+ | TIMESERIES(mytime,'day') AS dayLevel,
+ | TIMESERIES(mytime,'hour') AS hourLevel,
+ | TIMESERIES(mytime,'minute') AS minuteLevel,
+ | TIMESERIES(mytime,'second') AS secondLevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'minute')='2016-02-23 09:01:00'
+ | GROUP BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ | ORDER BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ """.stripMargin)
+
+ checkExistence(df, true,
+ "2016-01-01 00:00:00",
+ "2016-02-01 00:00:00",
+ "2016-02-23 09:00:00",
+ "2016-02-23 09:01:00",
+ "2016-02-23 09:01:50",
+ "30"
+ )
+ }
+
+ test("test timeseries table selection 19: select with many GROUP BY AND many filter") {
+ val df = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'year') AS yearLevel,
+ | TIMESERIES(mytime,'month') AS monthLevel,
+ | TIMESERIES(mytime,'day') AS dayLevel,
+ | TIMESERIES(mytime,'hour') AS hourLevel,
+ | TIMESERIES(mytime,'minute') AS minuteLevel,
+ | TIMESERIES(mytime,'second') AS secondLevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE
+ | TIMESERIES(mytime,'second')='2016-02-23 09:01:50' and
+ | TIMESERIES(mytime,'minute')='2016-02-23 09:01:00' and
+ | TIMESERIES(mytime,'hour')='2016-02-23 09:00:00' and
+ | TIMESERIES(mytime,'month')='2016-02-01 00:00:00' and
+ | TIMESERIES(mytime,'year')='2016-01-01 00:00:00'
+ | GROUP BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ | ORDER BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ """.stripMargin)
+
+ checkExistence(df, true,
+ "2016-01-01 00:00:00",
+ "2016-02-01 00:00:00",
+ "2016-02-23 09:00:00",
+ "2016-02-23 09:01:00",
+ "2016-02-23 09:01:50",
+ "30"
+ )
+ }
+
+ test("test timeseries table selection 20: filter < AND >") {
--- End diff --
Filter scenarios are already covered in TestTimeSeriesMatchStrategySuite.scala. Please remove duplicate scenarios
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2397/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
need rebase again
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3137/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1857/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174058262
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -82,149 +92,209 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
}
+ override def afterEach(): Unit = {
+ dropDataMaps("mainTable", "agg1_second", "agg1_minute",
+ "agg1_hour", "agg1_day", "agg1_month", "agg1_year")
+ }
+
test("test timeseries create table 1") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_second"), true, "maintable_agg0_second")
- sql("drop datamap agg0_second on table mainTable")
+ sql("DROP DATAMAP agg0_second ON TABLE mainTable")
}
test("test timeseries create table 2") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_hour"), true, "maintable_agg0_hour")
- sql("drop datamap agg0_hour on table mainTable")
+ sql("DROP DATAMAP agg0_hour ON TABLE mainTable")
}
+
test("test timeseries create table 3") {
checkExistence(sql("DESCRIBE FORMATTED maintable_agg0_day"), true, "maintable_agg0_day")
- sql("drop datamap agg0_day on table mainTable")
+ sql("DROP DATAMAP agg0_day ON TABLE mainTable")
}
+
test("test timeseries create table 4") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_month"), true, "maintable_agg0_month")
- sql("drop datamap agg0_month on table mainTable")
+ sql("DROP DATAMAP agg0_month ON TABLE mainTable")
}
+
test("test timeseries create table 5") {
checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0_year"), true, "maintable_agg0_year")
- sql("drop datamap agg0_year on table mainTable")
+ sql("DROP DATAMAP agg0_year ON TABLE mainTable")
}
- test("test timeseries create table 6") {
- intercept[Exception] {
+ test("test timeseries create table 6: TIMESERIES should define time granularity") {
+ sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
- s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'SEC_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ s"""CREATE DATAMAP agg0_second ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'SEC_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
""".stripMargin)
}
+ assert(e.getMessage.contains("TIMESERIES should define time granularity"))
}
- test("test timeseries create table 7") {
- intercept[Exception] {
+ test("test timeseries create table 7: Granularity only support 1") {
+ sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ val e = intercept[MalformedDataMapCommandException] {
sql(
s"""
| CREATE DATAMAP agg0_second ON TABLE mainTable
| USING '$timeSeries'
| DMPROPERTIES (
| 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='2')
+ | 'DAY_GRANULARITY'='1.5')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
+ assert(e.getMessage.equals("Granularity only support 1"))
}
- test("test timeseries create table 8") {
- intercept[Exception] {
+ test("test timeseries create table 8: Granularity only support 1") {
+ dropDataMaps("mainTable", "agg1_hour")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
- s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
+ s"""CREATE DATAMAP agg1_hour ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='hour=-2')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ }
+ assert(e.getMessage.contains("Granularity only support "))
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), false, "maintable_agg1_hour")
+ }
+
+ test("test timeseries create table 9: SECOND_GRANULARITY is null") {
+ sql("DROP DATAMAP IF EXISTS agg1 ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
- s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
+ s"""CREATE DATAMAP agg0_hour ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ }
+ assert(e.getMessage.contains("Granularity only support 1"))
+ }
+
+ test("test timeseries create table 10: Table already exists in database") {
+ val e = intercept[MalformedDataMapCommandException] {
+ sql(
+ s"""CREATE DATAMAP agg1_hour ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ sql(
+ s"""CREATE DATAMAP agg1_hour ON TABLE mainTable USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
}
+ assert(e.getMessage.contains(
+ "DataMap name 'agg1_hour' already exist"))
}
- test("test timeseries create table 9") {
- intercept[Exception] {
+ test("test timeseries create table 11: don't support create timeseries table on non timestamp") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
+ | CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING '$timeSeries'
| DMPROPERTIES (
| 'EVENT_TIME'='name',
- | 'SECOND_GRANULARITY'='1')
+ | 'HOUR_GRANULARITY'='1')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
+ assert(e.getMessage.equals("Timeseries event time is only supported on Timestamp column"))
}
- test("test timeseries create table 10") {
- intercept[Exception] {
+ test("test timeseries create table 12: Time series column dataTime does not exists in select") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
sql(
s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
+ | CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING '$timeSeries'
| DMPROPERTIES (
- | 'EVENT_TIME'='name',
- | 'SECOND_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
+ | 'EVENT_TIME'='dataTime',
+ | 'HOUR_GRANULARITY'='1')
+ | AS SELECT name, SUM(age) FROM mainTable
+ | GROUP BY name
+ """.stripMargin)
+ }
+ assert(e.getMessage.equals("Time series column dataTime does not exists in select"))
+ }
+
+ test("test timeseries create table 13: don't support create timeseries table on non timestamp") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
+ val e = intercept[MalformedCarbonCommandException] {
+ sql(
+ s"""CREATE DATAMAP agg0_hour ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='name',
+ | 'HOUR_GRANULARITY'='1')
+ |AS SELECT name, SUM(age) FROM mainTable
+ |GROUP BY name
+ """.stripMargin)
}
+ assert(e.getMessage.contains("Timeseries event time is only supported on Timestamp column"))
}
- test("test timeseries create table 11: USING") {
+ test("test timeseries create table 14: USING") {
val e: Exception = intercept[MalformedDataMapCommandException] {
sql(
- """CREATE DATAMAP agg1 ON TABLE mainTable
+ """CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING 'abc'
| DMPROPERTIES (
| 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
+ | 'HOUR_GRANULARITY'='1')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.equals("DataMap 'abc' not found"))
}
- test("test timeseries create table 12: USING and catch MalformedCarbonCommandException") {
+ test("test timeseries create table 15: USING and catch MalformedCarbonCommandException") {
val e: Exception = intercept[MalformedCarbonCommandException] {
sql(
- """CREATE DATAMAP agg1 ON TABLE mainTable
+ """CREATE DATAMAP agg0_hour ON TABLE mainTable
| USING 'abc'
| DMPROPERTIES (
| 'EVENT_TIME'='dataTime',
- | 'SECOND_GRANULARITY'='1')
+ | 'HOUR_GRANULARITY'='1')
| AS SELECT dataTime, SUM(age) FROM mainTable
| GROUP BY dataTime
""".stripMargin)
}
assert(e.getMessage.equals("DataMap 'abc' not found"))
}
- test("test timeseries create table 13: Only one granularity level can be defined 1") {
- sql("DROP DATAMAP IF EXISTS agg0_second ON TABLE mainTable")
+ test("test timeseries create table 16: Only one granularity level can be defined 1") {
+ sql("DROP DATAMAP IF EXISTS agg0_hour ON TABLE mainTable")
val e: Exception = intercept[MalformedCarbonCommandException] {
sql(
s"""
- | CREATE DATAMAP agg0_second ON TABLE mainTable
+ | CREATE DATAMAP agg0_hour ON TABLE mainTable
--- End diff --
why change just the name of the table??
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4291/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3475/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174127189
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesDropSuite.scala ---
@@ -16,15 +16,28 @@
*/
package org.apache.carbondata.integration.spark.testsuite.timeseries
+import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.test.util.QueryTest
import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
-import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
+import org.apache.carbondata.common.exceptions.sql.{MalformedCarbonCommandException, NoSuchDataMapException}
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.spark.exception.ProcessMetaDataException
class TestTimeSeriesDropSuite extends QueryTest with BeforeAndAfterAll with BeforeAndAfterEach {
+ val timeSeries = "timeseries"
+ var timestampFormat: String = _
--- End diff --
@sraghunandan suggested should not change the CarbonCommonConstants values after running the test case, because other place maybe need this constants value.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2979/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3386/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3614/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3110/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3438/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3360/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174064540
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesDataLoad.scala ---
@@ -330,16 +366,232 @@ class TestTimeseriesDataLoad extends QueryTest with BeforeAndAfterAll {
""".stripMargin)
checkAnswer(sql("select * from maintable_agg0_second"),
- Seq(Row(Timestamp.valueOf("2016-02-23 01:01:30.0"), 10),
- Row(Timestamp.valueOf("2016-02-23 01:01:40.0"), 20),
- Row(Timestamp.valueOf("2016-02-23 01:01:50.0"), 30),
- Row(Timestamp.valueOf("2016-02-23 01:02:30.0"), 40),
- Row(Timestamp.valueOf("2016-02-23 01:02:40.0"), 50),
- Row(Timestamp.valueOf("2016-02-23 01:02:50.0"), 50)))
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50)))
+ }
+
+ test("test timeseries table selection 14: load data into mainTable after create timeseries datamap ON TABLE and SELECT sub table") {
+ sql("DROP TABLE IF EXISTS main_table")
+ sql(
+ """
+ | CREATE TABLE main_table(
+ | mytime timestamp,
+ | name string,
+ | age int)
+ | STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+ sql(
+ s"""
+ | CREATE DATAMAP agg0_second ON TABLE main_table
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ | 'event_time'='mytime',
+ | 'SECOND_GRANULARITY'='1')
+ | AS SELECT mytime, SUM(age)
+ | FROM main_table
+ | GROUP BY mytime""".stripMargin)
+
+
+ sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' INTO TABLE main_table")
+
+ checkAnswer(sql("SELECT * FROM main_table_agg0_second"),
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:01:30.0"), 10),
+ Row(Timestamp.valueOf("2016-02-23 09:01:40.0"), 20),
+ Row(Timestamp.valueOf("2016-02-23 09:01:50.0"), 30),
+ Row(Timestamp.valueOf("2016-02-23 09:02:30.0"), 40),
+ Row(Timestamp.valueOf("2016-02-23 09:02:40.0"), 50),
+ Row(Timestamp.valueOf("2016-02-23 09:02:50.0"), 50)))
+ }
+
+ test("test timeseries table selection 15: load data into main_table after create timeseries datamap ON TABLE 1") {
--- End diff --
this scenario is already covered in the previous tests
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3474/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2271/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest sdv please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3355/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3355/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3636/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3846/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174123968
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala ---
@@ -354,21 +390,137 @@ class TestTimeSeriesCreateTable extends QueryTest with BeforeAndAfterAll {
checkExistence(sql("DESC FORMATTED mainTable_agg1"), true, "maintable_age_sum")
}
+ test("test timeseries create table 32: should support if not exists, create when same table not exists") {
+ sql("DROP DATAMAP IF EXISTS agg1_year ON TABLE mainTable")
+ sql(
+ s"""
+ |CREATE DATAMAP if not exists agg1_year ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ | 'event_time'='dataTime',
+ | 'YEAR_GRANULARITY'='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+ """.stripMargin)
+ checkExistence(sql("SHOW DATAMAP ON TABLE mainTable"), true, "agg1_year")
+ checkExistence(sql("DESC FORMATTED mainTable_agg1_year"), true, "maintable_age_sum")
+ }
+
test("test timeseries create table 20: don't support 'create datamap if exists'") {
val e: Exception = intercept[AnalysisException] {
sql(
s"""CREATE DATAMAP IF EXISTS agg2 ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'MONTH_GRANULARITY'='1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
+ | USING '$timeSeries'
--- End diff --
ok
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2947/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3047/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3223/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
rebase again
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1989/
---
[GitHub] carbondata issue #1856: [WIP][CARBONDATA-2073][CARBONDATA-1516][Tests] Add t...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3201/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1856
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3299/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest sdv please
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4224/
---
[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1856#discussion_r174129743
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesTableSelection.scala ---
@@ -99,124 +113,763 @@ class TestTimeseriesTableSelection extends QueryTest with BeforeAndAfterAll {
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/timeseriestest.csv' into table mainTable")
}
- test("test PreAggregate table selection 1") {
- val df = sql("select mytime from mainTable group by mytime")
+ test("test timeseries table selection 1") {
+ val df = sql("SELECT mytime FROM mainTable GROUP BY mytime")
preAggTableValidator(df.queryExecution.analyzed, "maintable")
}
- test("test PreAggregate table selection 2") {
- val df = sql("select timeseries(mytime,'hour') from mainTable group by timeseries(mytime,'hour')")
+ test("test timeseries table selection 2") {
+ val df = sql("SELECT TIMESERIES(mytime,'hour') FROM mainTable GROUP BY TIMESERIES(mytime,'hour')")
preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 3") {
- val df = sql("select timeseries(mytime,'milli') from mainTable group by timeseries(mytime,'milli')")
- preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ test("test timeseries table selection 3: No enum constant MILLI") {
+ val e = intercept[Exception] {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'milli')
+ | FROM mainTable
+ | GROUP BY TIMESERIES(mytime,'milli')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ df.show()
+ }
+ assert(e.getMessage.contains(
+ "No enum constant org.apache.carbondata.core.preagg.TimeSeriesFunctionEnum.MILLI"))
}
- test("test PreAggregate table selection 4") {
- val df = sql("select timeseries(mytime,'year') from mainTable group by timeseries(mytime,'year')")
+ test("test timeseries table selection 4") {
+ val df = sql("SELECT TIMESERIES(mytime,'year') FROM mainTable GROUP BY TIMESERIES(mytime,'year')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_year")
}
- test("test PreAggregate table selection 5") {
- val df = sql("select timeseries(mytime,'day') from mainTable group by timeseries(mytime,'day')")
+ test("test timeseries table selection 5") {
+ val df = sql("SELECT TIMESERIES(mytime,'day') FROM mainTable GROUP BY TIMESERIES(mytime,'day')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_day")
}
- test("test PreAggregate table selection 6") {
- val df = sql("select timeseries(mytime,'month') from mainTable group by timeseries(mytime,'month')")
+ test("test timeseries table selection 6") {
+ val df = sql("SELECT TIMESERIES(mytime,'month') FROM mainTable GROUP BY TIMESERIES(mytime,'month')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_month")
}
- test("test PreAggregate table selection 7") {
- val df = sql("select timeseries(mytime,'minute') from mainTable group by timeseries(mytime,'minute')")
+ test("test timeseries table selection 7") {
+ val df = sql("SELECT TIMESERIES(mytime,'minute') FROM mainTable GROUP BY TIMESERIES(mytime,'minute')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_minute")
}
- test("test PreAggregate table selection 8") {
- val df = sql("select timeseries(mytime,'second') from mainTable group by timeseries(mytime,'second')")
+ test("test timeseries table selection 8") {
+ val df = sql("SELECT TIMESERIES(mytime,'second') FROM mainTable GROUP BY TIMESERIES(mytime,'second')")
preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_second")
}
- test("test PreAggregate table selection 9") {
- val df = sql("select timeseries(mytime,'hour') from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 9") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 10") {
- val df = sql("select timeseries(mytime,'hour') from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 10") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
+ }
+
+ test("test timeseries table selection 11") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour'),SUM(age)
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 11") {
- val df = sql("select timeseries(mytime,'hour'),sum(age) from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 12") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour') AS hourlevel,SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
}
- test("test PreAggregate table selection 12") {
- val df = sql("select timeseries(mytime,'hour')as hourlevel,sum(age) as sum from mainTable where timeseries(mytime,'hour')='x' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable_agg0_hour")
+ test("test timeseries table selection 13") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')as hourlevel,SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='x' AND name='vishal'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ | ORDER BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
}
- test("test PreAggregate table selection 13") {
- val df = sql("select timeseries(mytime,'hour')as hourlevel,sum(age) as sum from mainTable where timeseries(mytime,'hour')='x' and name='vishal' group by timeseries(mytime,'hour') order by timeseries(mytime,'hour')")
- preAggTableValidator(df.queryExecution.analyzed,"maintable")
+ test("test timeseries table selection 14: TIMESERIES(mytime,'hour') match") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='2016-02-23 09:00:00'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
+ checkAnswer(df, Row(Timestamp.valueOf("2016-02-23 09:00:00.0")))
}
- test("test timeseries table selection 14: Granularity only support 1 and throw Exception") {
- val e = intercept[MalformedCarbonCommandException] {
- sql(
- s"""
- | CREATE DATAMAP agg3_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'HOUR_GRANULARITY'='2')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
- }
- assert(e.getMessage.contains("Granularity only support 1"))
- }
-
- test("test timeseries table selection 15: Granularity only support 1 and throw Exception") {
- val e = intercept[MalformedCarbonCommandException] {
- sql(
- s"""
- | CREATE DATAMAP agg3_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'HOUR_GRANULARITY'='1.5')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
- }
- assert(e.getMessage.contains("Granularity only support 1"))
- }
-
- test("test timeseries table selection 16: Granularity only support 1 and throw Exception") {
- val e = intercept[MalformedCarbonCommandException] {
- sql(
- s"""
- | CREATE DATAMAP agg3_second ON TABLE mainTable
- | USING '$timeSeries'
- | DMPROPERTIES (
- | 'EVENT_TIME'='dataTime',
- | 'HOUR_GRANULARITY'='-1')
- | AS SELECT dataTime, SUM(age) FROM mainTable
- | GROUP BY dataTime
- """.stripMargin)
- }
- assert(e.getMessage.contains("Granularity only support 1"))
+ test("test timeseries table selection 15: TIMESERIES(mytime,'hour') not match") {
+ val df = sql(
+ """
+ | SELECT TIMESERIES(mytime,'hour')
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'hour')='2016-02-23 09:01:00'
+ | GROUP BY TIMESERIES(mytime,'hour')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_hour")
+ checkExistence(df, false, "2016-02-23 09:00:00", "2016-02-23 09:01:00")
+ }
+
+ test("test timeseries table selection 16: TIMESERIES(mytime,'minute') match") {
+ checkExistence(sql("SELECT * FROM mainTable"), true,
+ "2016-02-23 09:01:30", "2016-02-23 09:02:40")
+ checkExistence(sql("SELECT * FROM mainTable"), false,
+ "2016-02-23 09:02:00", "2016-02-23 09:01:00")
+ val df = sql(
+ """
+ |SELECT TIMESERIES(mytime,'minute')
+ |FROM mainTable
+ |GROUP BY TIMESERIES(mytime,'minute')
+ """.stripMargin)
+ preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0_minute")
+ checkExistence(df, true, "2016-02-23 09:02:00", "2016-02-23 09:01:00")
+ checkAnswer(df,
+ Seq(Row(Timestamp.valueOf("2016-02-23 09:02:00.0")),
+ Row(Timestamp.valueOf("2016-02-23 09:01:00.0"))))
+
+ val df2 = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'minute')as minutelevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'minute')='2016-02-23 09:01:00'
+ | GROUP BY TIMESERIES(mytime,'minute')
+ | ORDER BY TIMESERIES(mytime,'minute')
+ """.stripMargin)
+ preAggTableValidator(df2.queryExecution.analyzed, "maintable_agg0_minute")
+ checkAnswer(df2, Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 60)))
+ }
+
+ test("test timeseries table selection 17: TIMESERIES(mytime,'minute') not match pre agg") {
+ val df = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'minute')as minutelevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'minute')='2016-02-23 09:01:00' AND name='vishal'
+ | GROUP BY TIMESERIES(mytime,'minute')
+ | ORDER BY TIMESERIES(mytime,'minute')
+ """.stripMargin)
+ checkAnswer(df, Seq(Row(Timestamp.valueOf("2016-02-23 09:01:00"), 10)))
+ preAggTableValidator(df.queryExecution.analyzed, "maintable")
+ }
+
+ test("test timeseries table selection 18: select with many GROUP BY AND one filter") {
+ val df = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'year') AS yearLevel,
+ | TIMESERIES(mytime,'month') AS monthLevel,
+ | TIMESERIES(mytime,'day') AS dayLevel,
+ | TIMESERIES(mytime,'hour') AS hourLevel,
+ | TIMESERIES(mytime,'minute') AS minuteLevel,
+ | TIMESERIES(mytime,'second') AS secondLevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE TIMESERIES(mytime,'minute')='2016-02-23 09:01:00'
+ | GROUP BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ | ORDER BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ """.stripMargin)
+
+ checkExistence(df, true,
+ "2016-01-01 00:00:00",
+ "2016-02-01 00:00:00",
+ "2016-02-23 09:00:00",
+ "2016-02-23 09:01:00",
+ "2016-02-23 09:01:50",
+ "30"
+ )
+ }
+
+ test("test timeseries table selection 19: select with many GROUP BY AND many filter") {
+ val df = sql(
+ """
+ | SELECT
+ | TIMESERIES(mytime,'year') AS yearLevel,
+ | TIMESERIES(mytime,'month') AS monthLevel,
+ | TIMESERIES(mytime,'day') AS dayLevel,
+ | TIMESERIES(mytime,'hour') AS hourLevel,
+ | TIMESERIES(mytime,'minute') AS minuteLevel,
+ | TIMESERIES(mytime,'second') AS secondLevel,
+ | SUM(age) AS SUM
+ | FROM mainTable
+ | WHERE
+ | TIMESERIES(mytime,'second')='2016-02-23 09:01:50' and
+ | TIMESERIES(mytime,'minute')='2016-02-23 09:01:00' and
+ | TIMESERIES(mytime,'hour')='2016-02-23 09:00:00' and
+ | TIMESERIES(mytime,'month')='2016-02-01 00:00:00' and
+ | TIMESERIES(mytime,'year')='2016-01-01 00:00:00'
+ | GROUP BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ | ORDER BY
+ | TIMESERIES(mytime,'year'),
+ | TIMESERIES(mytime,'month'),
+ | TIMESERIES(mytime,'day'),
+ | TIMESERIES(mytime,'hour'),
+ | TIMESERIES(mytime,'minute'),
+ | TIMESERIES(mytime,'second')
+ """.stripMargin)
+
+ checkExistence(df, true,
+ "2016-01-01 00:00:00",
+ "2016-02-01 00:00:00",
+ "2016-02-23 09:00:00",
+ "2016-02-23 09:01:00",
+ "2016-02-23 09:01:50",
+ "30"
+ )
+ }
+
+ test("test timeseries table selection 20: filter < AND >") {
--- End diff --
different purpose, but I can remove this.
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1856
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3429/
---
[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1856
retest this please
---