You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2018/03/20 08:29:58 UTC
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/2081
[CARBONDATA-2262] Support the syntax of 'using CARBONDATA' to create table
Add new function to Support the syntax of 'using CARBONDATA' to create table, for example:
`CREATE TABLE src_carbondata1(key INT, value STRING) using CarbonData OPTIONS('tableName'='src_carbondata1')`
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata CARBONDATA-2262UsingCarbonData
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2081.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2081
----
----
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3399/
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177397284
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/UsingCarbondataSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sql.commands
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterEach
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+
+class UsingCarbondataSuite extends QueryTest with BeforeAndAfterEach {
+ override def beforeEach(): Unit = {
+ sql("DROP TABLE IF EXISTS src_carbondata1")
+ sql("DROP TABLE IF EXISTS tableSize3")
+ }
+
+ override def afterEach(): Unit = {
+ sql("DROP TABLE IF EXISTS src_carbondata1")
+ sql("DROP TABLE IF EXISTS tableSize3")
+ }
+
+ test("CARBONDATA-2262: test check results of table with complex data type and bucketing") {
+ sql("drop table if exists create_source")
+ sql("create table create_source(intField int, stringField string, complexField array<int>) " +
+ "USING CARBONDATA OPTIONS('tableName'='create_source')")
--- End diff --
ok, done
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
retest this please
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4136/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4651/
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177285600
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/commands/SparkSessionForUsingCarbonDataExample.scala ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples.commands
+
+import java.io.File
+
+import org.apache.commons.io.FileUtils
+import org.apache.spark.sql.SparkSession
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+object SparkSessionForUsingCarbonDataExample {
--- End diff --
Don't suggest adding a new example, can add the new usage into the current example:SparkSessionExample
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4654/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4161/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3394/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2081
LGTM
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177399594
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/commands/SparkSessionForUsingCarbonDataExample.scala ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples.commands
+
+import java.io.File
+
+import org.apache.commons.io.FileUtils
+import org.apache.spark.sql.SparkSession
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+object SparkSessionForUsingCarbonDataExample {
--- End diff --
OK, done
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
@jackylk Please review it.
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177132703
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/UsingCarbondataSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sql.commands
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterEach
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+
+class UsingCarbondataSuite extends QueryTest with BeforeAndAfterEach {
+ override def beforeEach(): Unit = {
+ sql("DROP TABLE IF EXISTS src_carbondata1")
+ sql("DROP TABLE IF EXISTS tableSize3")
+ }
+
+ override def afterEach(): Unit = {
+ sql("DROP TABLE IF EXISTS src_carbondata1")
+ sql("DROP TABLE IF EXISTS tableSize3")
+ }
+
+ test("CARBONDATA-2262: test check results of table with complex data type and bucketing") {
+ sql("drop table if exists create_source")
+ sql("create table create_source(intField int, stringField string, complexField array<int>) " +
+ "USING CARBONDATA OPTIONS('tableName'='create_source')")
--- End diff --
no need to give tableName option, please modify
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
retest this please
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4122/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3217/
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177132855
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala ---
@@ -186,15 +186,17 @@ class DDLStrategy(sparkSession: SparkSession) extends SparkStrategy {
ExecutedCommandExec(CarbonResetCommand()) :: Nil
case org.apache.spark.sql.execution.datasources.CreateTable(tableDesc, mode, None)
if tableDesc.provider.get != DDLUtils.HIVE_PROVIDER
- && tableDesc.provider.get.equals("org.apache.spark.sql.CarbonSource") =>
+ && (tableDesc.provider.get.equals("org.apache.spark.sql.CarbonSource")
+ || tableDesc.provider.get.equalsIgnoreCase("CARBONDATA")) =>
--- End diff --
make it non capital
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4443/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4129/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4120/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
retest sdv please
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3424/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4450/
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177973231
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/SparkSessionExample.scala ---
@@ -63,123 +63,126 @@ object SparkSessionExample {
sparksession.sparkContext.setLogLevel("ERROR")
- // Create table
- sparksession.sql("DROP TABLE IF EXISTS sparksession_table")
- sparksession.sql(
- s"""
- | CREATE TABLE sparksession_table(
- | shortField SHORT,
- | intField INT,
- | bigintField LONG,
- | doubleField DOUBLE,
- | stringField STRING,
- | timestampField TIMESTAMP,
- | decimalField DECIMAL(18,2),
- | dateField DATE,
- | charField CHAR(5)
- | )
- | USING org.apache.spark.sql.CarbonSource
- | OPTIONS('DICTIONARY_INCLUDE'='dateField, charField',
- | 'dbName'='default', 'tableName'='sparksession_table')
+ for (keyWord <- Array("org.apache.spark.sql.CarbonSource", "carbondata")) {
--- End diff --
I think this makes it more complex, just use `carbondata` is enough
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
retest sdv please
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4619/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
retest sdv please
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2081#discussion_r177397743
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala ---
@@ -186,15 +186,17 @@ class DDLStrategy(sparkSession: SparkSession) extends SparkStrategy {
ExecutedCommandExec(CarbonResetCommand()) :: Nil
case org.apache.spark.sql.execution.datasources.CreateTable(tableDesc, mode, None)
if tableDesc.provider.get != DDLUtils.HIVE_PROVIDER
- && tableDesc.provider.get.equals("org.apache.spark.sql.CarbonSource") =>
+ && (tableDesc.provider.get.equals("org.apache.spark.sql.CarbonSource")
+ || tableDesc.provider.get.equalsIgnoreCase("CARBONDATA")) =>
--- End diff --
ok, done
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3965/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4162/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3210/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
@jackylk @chenliang613 CI pass, Please check it.
---
[GitHub] carbondata pull request #2081: [CARBONDATA-2262] Support the syntax of 'usin...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2081
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3969/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4128/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4144/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2081
retest sdv please
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4624/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2081
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3427/
---
[GitHub] carbondata issue #2081: [CARBONDATA-2262] Support the syntax of 'using CARBO...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2081
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3971/
---