You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/01/20 08:28:05 UTC

[GitHub] [carbondata] QiangCai opened a new pull request #3585: [DOC] CarbonExtension doc

QiangCai opened a new pull request #3585: [DOC] CarbonExtension doc
URL: https://github.com/apache/carbondata/pull/3585
 
 
    ### Why is this PR needed?
    
    
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] jackylk commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r368885129
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
+
+To enable CarbonExtensions in spark, we need to add the following configuration.
+
+|Key|Value|
+|---|---|
+|spark.sql.extensions|org.apache.spark.sql.CarbonExtensions|
+
+Start Spark SQL CLI by running the following command in the Spark directory:
+
+```
+./bin/spark-sql --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <carbondata assembly jar path>
+```
+###### Creating a Table
+
+```
+CREATE TABLE IF NOT EXISTS test_table (
 
 Review comment:
   Can you give more example of CREATE TABLE statement and explain the difference

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] chetandb commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
chetandb commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r368892051
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
 
 Review comment:
   It needs to be mentioned that this is supported only for Spark 2.3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r369349103
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
+
+To enable CarbonExtensions in spark, we need to add the following configuration.
+
+|Key|Value|
+|---|---|
+|spark.sql.extensions|org.apache.spark.sql.CarbonExtensions|
+
+Start Spark SQL CLI by running the following command in the Spark directory:
+
+```
+./bin/spark-sql --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <carbondata assembly jar path>
+```
+###### Creating a Table
+
+```
+CREATE TABLE IF NOT EXISTS test_table (
+  id string,
+  name string,
+  city string,
+  age Int)
+STORED AS carbondata;
+```
 
 Review comment:
   ok, I will list all syntax and the difference between SparkSession and CarbonSession.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r369353543
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] jackylk commented on issue #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#issuecomment-577025669
 
 
   LGTM

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] CarbonDataQA1 commented on issue #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#issuecomment-576193806
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1701/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r369353562
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
+
+To enable CarbonExtensions in spark, we need to add the following configuration.
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r369353650
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -99,6 +149,16 @@ val carbon = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(
    `SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("<carbon_store_path>", "<local metastore path>")`.
  - Data storage location can be specified by `<carbon_store_path>`, like `/carbon/data/store`, `hdfs://localhost:9000/carbon/data/store` or `s3a://carbon/data/store`.
 
+###### Option 2: Using SparkSession with CarbonExtensions
+Start Spark shell by running the following command in the Spark directory:
+
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] CarbonDataQA1 commented on issue #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#issuecomment-577006123
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1724/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] asfgit closed pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r369351294
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
+
+To enable CarbonExtensions in spark, we need to add the following configuration.
+
+|Key|Value|
+|---|---|
+|spark.sql.extensions|org.apache.spark.sql.CarbonExtensions|
+
+Start Spark SQL CLI by running the following command in the Spark directory:
+
+```
+./bin/spark-sql --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <carbondata assembly jar path>
+```
+###### Creating a Table
+
+```
+CREATE TABLE IF NOT EXISTS test_table (
 
 Review comment:
   fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] chetandb commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
chetandb commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r368904449
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
+
+To enable CarbonExtensions in spark, we need to add the following configuration.
+
+|Key|Value|
+|---|---|
+|spark.sql.extensions|org.apache.spark.sql.CarbonExtensions|
+
+Start Spark SQL CLI by running the following command in the Spark directory:
+
+```
+./bin/spark-sql --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <carbondata assembly jar path>
+```
+###### Creating a Table
+
+```
+CREATE TABLE IF NOT EXISTS test_table (
+  id string,
+  name string,
+  city string,
+  age Int)
+STORED AS carbondata;
+```
 
 Review comment:
   It can be mentioned that "Stored by" is not supported. Only "stored as" and "using" is supported.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] CarbonDataQA1 commented on issue #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#issuecomment-576979370
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1721/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] chetandb commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
chetandb commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r368902280
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -65,12 +65,62 @@ CarbonData can be integrated with Spark,Presto and Hive execution engines. The b
 #### Alluxio
 [CarbonData supports read and write with Alluxio](./alluxio-guide.md)
 
+## Installing and Configuring CarbonData to run locally with Spark SQL
+
+To enable CarbonExtensions in spark, we need to add the following configuration.
 
 Review comment:
   One or two points regarding how carbonextensions in spark will help the user can be added here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [carbondata] SachinR12 commented on a change in pull request #3585: [DOC] CarbonExtensions doc

Posted by GitBox <gi...@apache.org>.
SachinR12 commented on a change in pull request #3585: [DOC] CarbonExtensions doc
URL: https://github.com/apache/carbondata/pull/3585#discussion_r368913954
 
 

 ##########
 File path: docs/quick-start-guide.md
 ##########
 @@ -99,6 +149,16 @@ val carbon = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(
    `SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("<carbon_store_path>", "<local metastore path>")`.
  - Data storage location can be specified by `<carbon_store_path>`, like `/carbon/data/store`, `hdfs://localhost:9000/carbon/data/store` or `s3a://carbon/data/store`.
 
+###### Option 2: Using SparkSession with CarbonExtensions
+Start Spark shell by running the following command in the Spark directory:
+
 
 Review comment:
   Can also add this code for better understanding.
   SparkSession spark = SparkSession.builder().config(sc.getConf).enableHiveSupport.config("spark.sql.extensions","org.apache.spark.sql.CarbonInternalExtensions").getOrCreate();

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services